ABSTRACT. By Sharyl Stasser Wooton

Transcription

1 ABSTRACT DATA ENVELOPMENT ANALYSIS: A TOOL FOR SECONDARY EDUCATION RANKING AND BENCHMARKING By Sharyl Stasser Wooton Since the publishing of A Nation at Risk in 1983, the emerging trend in secondary education is to measure student, school and district performance using standardized proficiency exams. Districts in the same state are then ranked based on the percentage of students passing these exams. These rankings affect funding and public opinion about the district. Due to the No Child Left Behind Act being signed into Law in January 2002, the stakes based on these rankings will be even greater. All states must ensure that districts are making "adequate yearly progress", and if they are not, apply sanctions. The purpose of this study is to consider Data Envelopment Analysis (DEA) as an alternative to the current system for ranking and benchmarking districts. DEA is a variation of traditional linear programming and has proven to be a useful tool for obtaining relative efficiencies and developing peer groups of service oriented institutions.

2 DATA ENVELOPMENT ANALYSIS: A TOOL FOR SECONDARY EDUCATION RANKING AND BENCHMARKING A Thesis Submitted to the Faculty of Miami University in partial fulfillment of the requirements for the degree of Masters of Systems Analysis Department of Computer Science and Systems Analysis by Sharyl Stasser Wooton Miami University Oxford, Ohio 2003 Advisor Dr. Melanie Hatch Reader Dr. Donald Byrkett Reader Dr. Richard Hofmann

3 Table of Contents CHAPTER 1 INTRODUCTION 1 CHAPTER 2 REVIEW OF RELATIVE TOPICS STATE LEVEL PROFICIENCY EXAMS: CURRENT DISTRICT RANKING PROCEDURES EFFICIENCY: AN ALTERNATIVE MEASUREMENT FOR DISTRICT RANKING DATA ENVELOPMENT ANALYSIS: AN EXPLANATION DATA ENVELOPMENT ANALYSIS: MERITS IN RANKING DISTRICT PERFORMANCE DATA ENVELOPMENT ANALYSIS: DECOMPOSING EFFICIENCY 12 CHAPTER 3 METHODOLOGY PURPOSE DATA COLLECTION MODEL FORMULATION 18 CHAPTER 4 RESULTS AND DISCUSSION RANKING DISTRICTS DISTRICT IMPROVEMENT AND BENCHMARKING CONSIDERATIONS DECOMPOSING EFFICIENCY 24 CHAPTER 5 RECOMMENDATIONS AND CONCLUSION 26 APPENDIX A 28 EFFICIENCY SCORES AND PEER GROUPS 28 APPENDIX B 34 COMPARISON OF DMU EFFICIENCY SCORES AND DISTRICT RANKINGS 34 REFERENCES 35 ii

4 Acknowledgments I give thanks. To the students of Cincinnati Public Schools who taught me for ten years and developed in me a passion that I will always carry. To Dr. Melanie Hatch, my advisor, who expected the best of me, provided guidance in a fun way, and cared about this topic. To Dr. Donald Byrkett and Dr. Richard Hofmann, my committee members, who took the time and energy to make this a better thesis. To the "Calc II Pit Crew" comprised of: Tim, my committed husband and foundation, who made me laugh TJ, my adorable son, who provided perspective when I needed it Jayne, my loving Mother, who knew when I needed a cup of tea and John, my first programming teacher (and step-dad), who was always the voice of reason. To Dana, my "rational-shmational sister, who insisted I have a life outside of school. To Leslie, my kindred spirit, who understood me without explanation. To Josette, who cared for my other creative passion, TJ, so I could take time for this creation. To Bradley, Sun, Tom, Jon, Norm, Eric, Emily, Nellie, Ivan, and Trung, my fellow classmates, who understand all the stupid computer jokes and suffered alongside me in the "dungeon". To the wonderful staff of The Computer Science and System Analysis Department who are a great bunch of people. iii

5 Chapter 1 Introduction The United States population has been increasingly concerned about the quality of public education since the publishing of A Nation at Risk in For some, the answer to improving the quality of public education is to hold schools accountable for student learning by using standardized performance measures. Pass rates from standardized tests are being used as a performance measures with increased frequency to evaluate students, schools, and districts. In January of 2002, President Bush s education-reform bill was signed into law. The law will likely increase the number of states utilizing standardized (proficiency) exams on a state-wide basis. The law requires each state to create an accountability system that reports district performance. The accountability system produced in response to the law will lead to the rankings of districts and states. However, these rankings have been predominantly determined by the percentage of students passing one or more tests. A ranking system that is based on the percentage of students passing exams does not accurately reflect the transformational process of education. Students are the inputs to the transformational process and are analogous to raw materials in a manufacturing setting. The educational system adds value to the input (students) by teaching them various skills. Thus, students are transformed into productive members of society (output). However, students enter schools with different assets; schools are given different levels of support; and communities are involved at different levels. All of these factors are resources that a school uses to educate students or transform them to their new status. The amount of these resources is not considered when ranking is based solely on the percentage of students passing the exams. There is a linear programming application, Data Envelopment Analysis (DEA) that considers the relationship between these resources (inputs) and the students' performance on tests (outputs). DEA also provides benchmarking data in the form of peer groups to enable similar districts to learn from each other. The purpose of this thesis is to demonstrate an alternative way to measure performance of districts and states at the secondary education level that more accurately reflects the transformational process of education. 1

6 Chapter 2 Review of Relative Topics 2.1 State Level Proficiency Exams: Current District Ranking Procedures All fifty states have some type of statewide assessment tool in place for their public school systems. (Doherty, 2002) All states use these tools to measure student success and/or hold the districts accountable for student learning. According to Doherty, eight states use standardized tests to determine promotion, and seventeen states use them to determine high school graduation eligibility. Ideally, these tests are aligned with the state curriculum standards or learning outcomes and are often divided by subject. President Bush's education-reform law states that by all states must provide assessment measures in grades 3-8 and once in high school for the subjects math and reading. Later, science will be added as a requirement. Every state will be required to have an accountability system to report the chosen measures of performance. The states will apply sanctions or rewards to districts based on their yearly progress. (Fair Test Website, 2002) Furthermore, the law incorporates the National Assessment of Educational Progress to be used as a benchmarking tool to ensure that states do not set low performance standards. (Miller, 2002) The new law will require all states to set performance standards. Many states already use these proficiency scores (minimal competency) to determine students' eligibility to graduate and the district-level proficiency pass rates are used to rate district performances. For example, the State of Ohio classifies districts as "Excellent", "Effective", "Continuous Improvement", "Academic Watch" or "Academic Emergency". In the year , districts received rankings based on the number of standards met out of a possible twenty-seven standards. 2

7 Proficiency Exams Reading % Passed Writing % Passed Citizenship % Passed Math % Passed Science % Passed 4 th Grade 75 % 75 % 75 % 75 % 75 % 6 th Grade 75 % 75 % 75 % 75 % 75 % 9 th Grade( 8 th and 9 th Grade Results) 75 % 75 % 75 % 75 % 75 % 9 th Grade (8 th, 9 th and 10 th Grade Results) 85 % 85 % 85 % 85 % 85 % 12 th Grade 60 % 60 % 60 % 60 % 60 % Table 1: 25 of the 27 Standards for District Ranking in Ohio Table 1 shows that twenty-five of the standards are based on districts achieving a minimum passing rate for the proficiency exams at each grade level. The other two standards are a minimum of 93% attendance rate, and 90% graduation rate. Similarly, in many other states, districts are given a numerical or categorical rankings based on the number of overall students to pass the proficiency exams. (Education for the Information Age Website, 2002) The No Child Left Behind Act (NCLBA) guarantees that many more states will employ testing as a way to rate their school districts performances in order to apply rewards and sanctions. In fact, the states are mandated to develop accountability systems to ensure "adequate yearly progress" for districts. "Adequate yearly progress" is calculated using a base percentage from the year Each year until 2014, districts must improve by 12% of the difference between 100% and the base percentage. For example, a district with 40% passing rate in the year would have to have a 5% ( (100-40)/12 ) increase each year. Therefore, theoretically all districts would have 100% passing by the year (Fair Test Website, 2002) Districts that do not progress at this rate will receive federally mandated sanctions that may lead to restructuring of the school. Restructuring would mean one of the following options: reopen the school as a charter school, replace most of the current staff, or turn the operation of the school over to the state. None of these sanctions have been shown to improve test scores. (Fair Test Website, 2002) Furthermore, these sanctions are extremely disruptive to the students that the NCLBA is aiming to protect. 3

8 2.2 Efficiency: An Alternative Measurement for District Ranking Percentage of students passing exams is an inadequate measure for ranking performance especially since these exams will be used to apply rewards and sanctions to districts. These types of rankings ignore the transformational process of education. In addition, parameters sometimes used to rate districts such as student attendance rate can be considered an input into the educational process instead of an output measure. Furthermore, these types of rankings do not take into account the resources that a district may have or not have available. For instance, a district with 75% passing rate may be more efficient than a district with 90% passing rate given the available resources. Using pass rate is particularly dangerous because districts performing well in the transformational process may never be recognized because their exam pass rates are not high enough. However, the district with the lower pass rate may be discovering ways to effectively and efficiently educate students given their resources. Because the educational process is transformational in nature, the objective questions to be asked about districts' performances are related to the efficiency of the district. Ranking systems based solely on output measures like test scores are correct only if we assume equality of inputs. (Hatch et al., 2000) Clearly this is not the case for public schools. The ideal goal for lawmakers and citizens is to see that districts are held accountable for the results they produce when educating students given their resources. Traditionally, manufacturing organizations which also transform a product from one state to another have used absolute efficiency measurements to hold production units accountable. However, determining absolute efficiency of decision making units (DMU) that deal with a physical product are much easier to compute than service organizations because quite often the inputs/outputs are physical counts of items. A DMU is considered efficient if the ratio of output produced to input consumed is greater than 1. (Hatch and Hill, 2001) The absolute efficiency of a DMU whose product is more abstract such as the learning of students is impossible to calculate. Since calculating absolute efficiency for service organizations is not possible, there are two other tools commonly used for comparing efficiencies. The first is Cost Benefit Analysis. This tool does not work well with educational organizations because it demands that inputs and outputs are expressed in monetary values. There are many 4

9 versions of Cost Benefit Analysis that lessen these constraints. However, the best solution requires weights of decision variables to be set a priori which leads to analyst bias. (Julnes, 2000) Regression Analysis defines an equation that represents expected outputs given specific levels of inputs. Then, these expected outputs are compared to actual outputs as a measurement for performance. The problem with this tool is that the equation determined is based off the average unit under evaluation. Therefore, there is no clear analysis of the level of productivity improvement that could be accomplished since expected outcome is based off an average. (Julnes, 2000) Regression Analysis sets the performance standard lower than some DMU's are able to achieve, and those units are not recognized. Over the last two decades, Data Envelopment Analysis (DEA) has proven to be a valuable relative efficiency tool for ranking service industry units. Authors have demonstrated that DEA overcomes some of the pitfalls shown in other efficiency measurement techniques. Conceição, Portela, and Thanassoulis (2001) used it to identify the root sources of a pupil s under attainment at school. Sarrico, Hogan, Dyson, and Thanassopoulos (1997) used DEA to produce customized individual league tables to inform the potential student in his/her choice for higher education institutions. In practice, DEA has been used by Shim and Kantor to develop peer groups of academic research libraries. Sherman and Ladino (1995) used DEA to analyze a 33 branch banking system, and recommend changes to non-efficient branches based on peer groups". 2.3 Data Envelopment Analysis: An Explanation The Data Envelopment Analysis methodology developed by Charnes, Cooper and Rhodes (1978) is a modification of traditional linear programming (LP). LP is a mathematical model that has two basic components. First, the goal is to maximize (or minimize) some objective function such as profit. The objective is expressed as a linear function which contains the decision variables and is called the objective function. Secondly, there are constraints that limit the degree to which the objective can be met 5

10 which are also expressed as linear functions. A further explanation of LP can be found by Anderson, Sweeney and William (1997). DEA is a special type of LP that calculates relative efficiency of DMU's. Relative efficiency of the j th DMU is determined by the weighted ratio, e j u1 y = v x 1 j 1 1 j + u + v 2 2 y x 2 j 2 j + + u + + v s r y x rj sj (1) where r is the total number of outputs, s is the total number of inputs, y mj is the m th output of the j th unit, x ij is the i th input of the j th unit and u m and v i are the weights. (Hatch et al., 2002) The highest scoring DMU then is considered the most efficient unit and all others are rated in comparison to this unit. (Hatch et al., 2002) It is important to note that relative efficiency compares all DMU s within a group and does not determine an absolute efficiency. It is conceivable that all DMU s could be found inefficient if an absolute efficiency could be determined. As mentioned above, relative efficiency is used because determining absolute efficiency for service-oriented organizations is not possible. In equation 1, the weights are decision variables and represent the worth of inputs and outputs. Traditionally, when calculating relative efficiency based strictly on this equation, the decisions for the values of the weights are arbitrarily set by the analyst and can lead to biases against DMU's. For example, Hatch et al. reviewed the US News and World Report algorithm (UNA) for ranking colleges and universities. The decision variables and weights in the UNA are Reputation (0.25), Graduation and Retention Rate (0.20), Faculty Resources (0.20), Selectivity (0.15), Financial Resources (0.10), Alumni Giving (0.05) and Graduate Rate Performance (0.05). In UNA, the analysts decide a priori the weights of the decision variables. Hatch et al. compared UNA to two other methodologies, Alternative Cross-Efficiency (AXE) Ratings and DEA using a sample of public universities. The weights were different from the UNA weights when using these methodologies. "The average of the weights computed by the AXE theoretically reflects resource worth that might be assigned by the average decision-maker. The DEA weights 6

11 are chosen in such a way to make each university as attractive as possible." (Hatch et al, 2002). DEA solves the problem of analyst bias where weights are determined a priori. The conceptual formulation of DEA is as follows: Max s.t. e k = r um ymk (2) m= 1 s vi xik i= 1 r um y (3) j = 1... n mj m= 1 s vi x i= 1 ij 1 u, ε (4) m = 1... r, i = 1... s m v i Again, the weights are decision variables in a DEA problem, but the weights are derived by the data set. The DEA formulation for each DMU k determines weights (u m and v i ) for inputs and outputs that would maximize efficiency of that DMU k (equation 2). Equation 3 limits the maximum efficiency of each DMU to 100%. The value of each weight is restricted in the problem so that no weight can be 0 %( equation 4). Because the objective is to maximize efficiency, the LP formulation would force weights to zero for any input/output that would reduce efficiency. If equation 4 was removed, DEA would not detect inefficiencies in regards to inputs/outputs whose respective weights are zero. The conceptual formulation described above is a set of non-linear functions. Using algebraic manipulation, the conceptual formulation can be changed to all linear expressions and written as follows: Max e k = r m = 1 u m y mk (5) s.t. s i= 1 s i= 1 v = 1 (6) i x ik r v x u y 0 (7) j= 1... n i ij m= 1 m mj u, ε (8) m = 1... r, i = 1... s m v i 7

12 For every DMU k under consideration, there is a DEA LP problem formulated. We do this so every DMU has the chance to be rated using equation 5, the objective function. The objective is to maximize efficiency of the DMU k. Notice that the weights for the inputs and outputs are the decision variables of the DEA problem. (Winston, 1994) One constraint placed on the problem is that the sum of the inputs for the DMU will be equal to one (equation 6). This is done to change the original non-linear formulation to one expressed in linear equations. The majority of the constraints (equation 7) are applied to ensure that no DMU can be greater than 100% efficient. For example, consider a DEA problem with twelve DMU's. There will be twelve separate DEA formulations, and each formulation will have twelve constraints (equation 7) that ensure each of the DMU's are less than or equal to 100% efficient. These constraints are the basis for benchmarking which will be discussed in the next section. Finally, analysts have added the last set of constraints (equation 8) to keep any decision variable from being set to zero. The formulation described above is the CCR (Charnes, Cooper and Rhodes) Input-Oriented Model. There are four basic DEA model formulations: CCR Input- Oriented, CCR Output-Oriented, BCC (Banker, Charnes and Cooper) Input-Oriented, and BCC Output-Oriented. These models can be classified based on two factors: 1) how the efficient frontier is produced and 2) how an inefficient DMU is projected onto the efficient frontier. The efficient frontier is set of DMU's which are considered to be efficient. The efficient frontier can be built assuming a constant returns-to-scale (CRS) or variable returns-to-scale (VRS). The CRS is described by Cooper et al. as "if an activity (x,y) is feasible, then, for every positive scalar t, the activity, (tx, ty) is also feasible." (2000) For example, as the number of tellers in a bank double; we would expect the number of services also double. The CCR model assumes a CRS. In the VRS case, the frontier is produced creating a convex hull with the outermost efficient DMU's. The efficient frontier is built by connecting these relatively efficient DMU's with linear segments. The name variable returns-to-scale results from the fact that these different linear segments may have decreasing, increasing, or constant returns-to-scale. For example, as the 8

13 number of teachers doubles in a school district, we would not expect to see the number of students graduating from school doubling. The second classification factor is how an inefficient DMU is "projected onto the frontier". In other words, the focus the model takes for making an inefficient DMU efficient. Model formulations can have an input or output orientation. These two formulations are the inverse of each other. Input orientation focuses on using fewer inputs to produce the same output while output orientation focuses on producing more output with the given inputs. Output oriented formulation is excellent for evaluating DMU's that have little control over the amount of input, but should seek to maximize their outputs given the level of input available. Lack of control over inputs happens in service organizations like school districts where they cannot determine the number of children on free and reduced lunch, their budget, etc. A further explanation of these models can be found by Cooper et al (2000). 2.4 Data Envelopment Analysis: Merits in Ranking District Performance DEA has many strengths that make it the best tool for deciding relative efficiency between districts and thereby determining rankings. The first is that multiple inputs and outputs of differing units can be used as long as the inputs and outputs are similar for each unit being compared. Therefore, we can use inputs such as class size, median income level of households, and outputs such as the percentage of students passing exams. Secondly, as described above, weights do not have to be arbitrarily set by analysts prior to evaluation. Another benefit of DEA is that it can be used as a benchmarking tool when comparing units. Table 2 illustrates an example DEA problem outlined by Hatch et al. (2000). This problem has one output and two inputs. Figure 1 is a graph of the output/input ratios from this example problem. Efficient DMU s (E, B, F, and C) determine the piecewise Linear Envelopment Surface (Charnes et al., 1995) of this DEA problem. Inefficient DMU's (D and A) solution points do not lie on this envelopment surface. However, these inefficient DMU s can be projected onto the envelopment surface. The algorithm for projection depends on the orientation of the DEA model, and the place of projection determines the peer groups for the inefficient 9

14 DMU. Projecting DMU D onto the surface, we see that D falls between DMU's B and F which make up D s peer group. The difference along the line between where DMU D is currently located and the projection on the surface (D ) is the inefficiency of DMU D. The efficiency score for DMU D is obtained by the ratio 0D/0D. DMU Input 1 Input 2 Output Output/Input 1 Output/Input 2 DEA Efficiency Score A B C D E F Table 2: Example DEA Problem Example DEA Problem E B D' Envelopment Surface: Line Segments EB, BF, FC Output/Input D F A' C A Inefficient DMU's Output/Input 1 Figure 1: Example DEA Problem Peer group determination can also be explained algebraically. In the maximization of any LP problem, the dual price is defined as "the improvement in the 10

15 value of the optimal solution per unit increase in the right-hand side of the constraint. (Anderson et al., 1997) In the DEA formulation, remember that equation 8 constrained each DMU from being greater than 100% efficient. The dual price for each DMU in DEA would be the amount of increase in the DMU k objective function for each unit of increase over the 100% (right hand side of the constraint). If a dual price is zero, then the constraint is considered "non-binding" which means it does not affect the feasible solution to the maximization problem. For example, a DMU j in equation 8 that never reached 100% efficiency would be "non-binding". However, a nonzero dual price indicates that a constraint is "binding" which means it was a limiting factor to the solution. In DEA, a "binding" DMU j would reach 100% efficiency before DMU k. Therefore, relaxing the right-hand side of this "binding" constraint would increase the feasible solution space which would then allow the objective function to increase. The nonzero dual values for the DMU s efficiency constraints indicate which DMU's are considered efficient in comparison to the evaluated unit, DMU k. These nonzero dual values show exactly where the evaluated unit could be expected to produce higher outputs and/or use less of the inputs. In order to further understand dual prices, consider the example DEA problem in Table 2 again. Table 3 illustrates the dual prices found in the solution to the DEA formulations for the inefficient DMU's D and A. DMU DMU D Dual Prices DMU A Dual Prices A 0 0 B C D 0 0 E 0 0 F Table 3: Dual Prices DMU D has nonzero dual prices for the constraints related to DMU's B and F. We can create D by averaging the output and input vectors of DMU B and F with their respective dual prices, and This is because graphically D lies on the line between DMU's B and F, specifically at the point which is of B and of F. 11

16 Input values of D : = Output values of D : The vectors [ ] +.668[ 30] [ ] = and 3.8 are the input vectors of DMU's B and F respectively from 12 the original problem (Table 2). Furthermore, the vectors [ 12.5] and [ 30 ] are the output vectors of DMU's B and F respectively. This algebraic analysis gives more detail about exactly where D is inefficient. Table 4 shows that D produces the same output with less of both Input 1 and Input 2 than D. The calculated output/input ratio values for D (1.736 and 2.604) place D on the envelopment surface in Figure 1. DMU Input 1 Input 2 Output Output/Input 1 Output/Input 2 D D Table 4: Comparison of D to D 2.5 Data Envelopment Analysis: Decomposing Efficiency Furthermore, DEA has also been used to evaluate student performance at the individual student level across several schools. (Conceição et al., 2001) The authors were interested in identifying the "root" cause of inefficiency. The objective was to determine if the student's lack of performance on standardized tests was a result of the school or the student. This is accomplished by decomposing an overall efficiency score into its component pieces using DEA. Student performance was computed at different levels of aggregation. First students were evaluated against their immediate peers within the same school. This allowed the authors to compute an efficiency score that related directly to student abilities. Then, the authors evaluated students against students within all the schools. This allowed for the computation of an efficiency score that related to the student as well as the individual school. The authors found the efficient boundary (envelopment surface) 12

17 for each school as well as an all-schools-efficiency boundary. Using DEA, they obtained two efficiencies: pupil-within-school efficiency (EFF wj0 ) and pupil-within-all-schools efficiency (EFF ij0 ). Then, a school-within-all-schools efficiency (EFF sj0 ) was calculated as EFF ij0 / EFF wj0. The authors used the following rules to identify whether the individual student or the school was more of an influence in inefficiency. If EFF sj0 is less than one and EFF wj0 = 1 (the student was 100% efficient within his/her school), then the inefficiency is attributed to the school. However, if EFF sj0 =1 and EFF wj0 is less than one, the inefficiency is attributed to the pupil. Next, schools were broken into types based on their funding source. Then, pupil-within-school-type efficient boundaries were determined in order to identify whether the type of school affected efficiency. This decomposing efficiency method was utilized in this research to look at district efficiencies within-state (EFF W ) and within-all-states (EFF A ). Then, a statewithin-state efficiency (EFF S ) was calculated as EFF A /EFF W. Figures 2-4 are simplified representations of a DEA problem using two outputs and one input for the sample DMU's. Envelopment surfaces were determined for State 1 (Figure 2) and State 2 (Figure 3) as well as an envelopment surface considering both states (Figure 4). The DMU represented by Point A has an EFF W = 1, EFF A < 1and EFF S < 1. Therefore, the State would be identified as the "root" cause of inefficiency. In contrast, the DMU represented by Point B has an EFF W < 1, EFF A < 1 and EFF S = 1, so the "root" cause of inefficiency is attributed to the district. 13

18 State 1 Envelopment Surface 1.05 Output 2/Input Point B EFF W < 1 Point A EFF W = 1 State Output 1/Input Figure 2: Example DEA: Decomposing Efficiency (district-within-state 1) State 2 Envelopment Surface Output 2/Input State Output 1/Input Figure 3: Example DEA: Decomposing Efficiency (district-within-state 2) 14

19 Combination Envelopment Surface 1.05 Output 2/Input Point B EFF A < 1 Point A EFF A < 1 State 1 State Output 1/Input Figure 4: Example DEA: Decomposing Efficiency (district-within-all-states) 15

20 Chapter 3 Methodology 3.1 Purpose The purpose of this thesis is to build a framework for the use of DEA as a means for ranking districts with regards to proficiency exam performance. This research is an extension of earlier work done by Hatch et al. (2000) where they examined DEA as a means for comparing school divisions within the state of Ohio. As a result of their research, some State Superintendents requested further study using relative efficiency as a performance measure. The research was conducted in four stages: 1) Districts were separated into their respective states and an efficiency score was calculated using DEA. This is analogous to EFF wj0 in the Conceição et al. paper. 2) Then all the districts were combined into a single data set. A new efficiency score was calculated for each district within all the states which is similar to EFF ij0. 3) DEA was then used in its capacity as a benchmarking tool to determine "peer groups". The "peer groups" were chosen from the population of districts that were used in stages 1 and 2 above. 4) Finally, whether the "root" cause of inefficiency lies at the district or state level was analyzed using the decomposition technique outlined in Conceição et al similar to EFF sj Data Collection Data was collected from five states (Georgia, Minnesota, Ohio, South Carolina and Texas) from the school year. States were chosen based on availability of data and similar testing policies. The states chosen all use obtainment of a minimal score on the proficiency exams as a component of their graduation requirements. All five states include at least a reading, writing, and math section of the proficiency exam. In addition, all these states have had their proficiency exam in place for at least seven years. One decision the analyst must make when using DEA is the number of DMU's to use in the comparison. There is no definitive rule about the absolute size of the sample, but it has been theorized that it should be at least 2(r x s) where r is the number of outputs and s in the number of inputs. This allows for enough distinction to be made between the districts and keeps all districts from possibly being considered efficient. (Dyson et al., 16

21 2001) Three outputs (test scores) that were common to all states have been selected. In addition, the maximum inputs would be limited to five in order to keep the overall formulation small in order to increase distinction between the districts.. Therefore, because of the within state analysis, a total of thirty districts were chosen from each state. District size was thought to have a significant effect on efficiency so a stratified random sample was conducted based on district total enrollment (Table 5). The breakdowns for the categories were chosen by creating an even distribution across the categories between all states. The divisions came from the natural breaking points in the data and each district was selected randomly from the available pool in that enrollment category. Enrollment Categories Number of Districts Per State < ,000-2, ,000-4, ,000-9, ,000 19,999 5 >20, Table 5: District Stratified Random Sample Once the districts to be included were determined the data was collected through each State Department of Education. The following data sets were analyzed to see if they should be included as inputs: district size, district annual expenditure per student, average attendance rate of teachers and students, percent of student and teachers by race and class, average salary of teachers, average experience of teacher, staff to student ratio, extracurricular involvement, educational level of parents and percent of students spending less than half the year in the district. The outputs analyzed were the math, reading, and writing proficiency exam passing rates at the district level. The final input set to be included was chosen using correlation analysis as a screening method. The district input factors that were available from all states were correlated to the three content area pass rates for each state. Table 7 shows the results for State 1. The inputs were selected so that they correlated well with the output measures but not with each other. When taking all the states into consideration, only two inputs had a correlation to all exam pass rates: total expenditure per student and percent 17

22 of students on free and reduced lunch. Although enrollment and average teacher experience correlated in state 1, they both had correlation scores lower than 0.10 in other states. Therefore, we did not use them in the final analysis. The inverse of percent of students on free and reduced lunch was used as an input since the actual resource for districts is the percent of students not on free and reduced lunch. Also, total expenditure per pupil was normalized on a scale of 1 to 100 so it is on the same scale as free and reduced lunch. OUT1 OUT2 OUT3 IN1 IN2 IN3 IN4 IN5 IN6 Math Scores Reading Scores Writing Scores Expenditures Per Pupil Average Teacher Salary Table 6: Input/Output Key for Table 7 Students Receive Free and Reduced Price Lunch (%) Students Per Teacher Average Teacher Experience Enrollment State 1 OUT1 OUT2 OUT3 IN1 IN2 IN3 IN4 IN5 IN6 OUT OUT OUT IN IN IN IN IN IN Table 7: State 1 Correlations 3.3 Model Formulation Banxia Frontier Analyst Professional (Jones and Tait, 1995) was used to perform the DEA analysis. There were two model choices to make. The first choice is whether the optimization mode will be input or output oriented. As described earlier, input oriented seeks to minimize inputs to produce the current output. The output oriented optimization mode seeks to maximize the outputs given existing inputs. Output oriented 18

23 optimization mode was chosen because districts do not have control over the amount of their inputs. The second choice was whether the scaling mode should be a constant or a variable returns-to-scale. As a reminder, constant returns-to-scale means that as input is doubled the amount of output is doubled. Whereas, variable returns-to-scale assumes that as an input increases there is not an equal increase in the amount of output produced. As the total expenditures per student or percent of students not on free and reduced lunch increase, the percent of students passing the proficiency tests increase at a slower rate and eventually fall off. Therefore, the best choice is variable returns-to-scale. These two model choices describe the BCC Output-Oriented model. The formulation for DMU k is expressed as, Min z k = s i= 1 v x i ik v k (9) s. t. s v x i ij i= 1 m= 1 r u m y mj v k 0 (10) j = 1.. n r m= 1 u m y mk = 1 (11) u, ε, v k free in sign (12) m = 1... r, i = 1... s m v i Although this is stated as a minimization problem, this formulation is considered the dual of the maximization of output problem. The dual of a problem in LP is an alternative way to solve the formulation where the same objective function values(s) are reached. For a detailed explanation of duality, see Winston (1994). Then, two categories of efficiencies were determined using DEA: district-withinstate (EFF W ), and district-within-all-states (EFF A ). During this analysis, peer groups on both the state and all state level were determined for inefficient districts. Finally, applying the technique of Conceição et al, a state-within-states efficiency (EFF S ) for each district was determined as EFF W / EFF A. Then, using a comparison between these efficiencies the "root" cause of inefficiency can be determined. As a last piece of analysis, an average EFF S was calculated for each state in order to compare states. 19

24 Chapter 4 Results and Discussion Each DMU was coded as X_NN. Where X is the state code, and NN is the district number. The data was coded to preserve the identities of the districts involved with the study. The final values from the DEA analysis are illustrated in the appendices of this document. Appendix A lists each DMU with the district-within-state (EFF W ), districtwithin-all-states (EFF A ), State-within-State(EFF S ), and DMU's that made up the peer groups. 4.1 Ranking Districts Because States 1 and 3 existing ranking systems are similar, they have been chosen to compare current rankings to DEA efficiency scores. The rankings categories for these states are represented by scores of 1=Excellent to 5=Unacceptable. Both states determine these ranking using a combination of pass rate on proficiency exams and two other factors. As discussed, DEA also provides efficiency scores that can be used for rankings. Figure 5 illustrates the relationship between the two rankings. The correlation is Since the correlation is positive, DEA ranks districts in the same general direction as the current system. So, DEA preserves the original ranking schema, but discrepancy between DEA and current systems show in the rankings of some districts. 20

25 Efficiency Score Current District Rating (1=Excellent to 5=Unacceptable) Figure 5: Comparison of DEA Efficiency Scores to Current District Rankings The important element of this comparison is the districts that change positions based on DEA versus current ranking techniques (Table 8). Some districts were identified as efficient although the state currently ranks them below other districts. The trend in these ranking improvements tended to be districts with exam pass rates slightly below the average for the state, but also with drastically less than average resources. (Table 9 illustrates these state averages). For instance, DMU 3_08 was 100% efficient but received a ranking of 4 from the state. DMU 3_08 had pass rates of 0.63 (Math), 0.75 (Reading), and 0.74 (Writing) compared to the state averages of (Math), (Reading) and (Writing). In addition, DMU 3_08 had an extremely low percent 0.03 (compared to the state average of 0.549) of students not on free and reduced lunch while their expenditures were above average (0.88 compared to 0.763) but not drastically. Other districts that were given a ranking of 1 were identified as relatively inefficient given their resources. (See Table 8) These DMU tended to have values high above the state average for most inputs and outputs. For example, DMU 1_14 has a ranking of 1, but received an efficiency score of 96.97%. DMU 1_14 had pass rates of 21

26 .86 (Math),0.97 (Reading), and 0.96(Writing) compared to the state averages of (Math), (Reading) and 0.897(Writing). In addition, DMU 3_08 had an high percent 0.94 compared to the state average of of students not of free and reduced lunch while their expenditures where above average (0.81 compared to 0.763) but not drastically. The Appendix B contains the ranking comparison of all districts. DMU District-within-state (EFF W ) Current Ranking by State 1_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Table 8: Comparison of DEA and Current Rankings in Two States Math Pass Rates Reading Pass Rates Writing Pass Rates Percent Students Not on Free and Reduced Lunch Expenditures Per Pupil Averages State State State State State All Table 9: Average Values for Inputs and Outputs 22

27 4.2 District Improvement and Benchmarking Considerations In order to interpret individual district-within-state results, consider the detailed analysis for DMU 4_03 as shown in Table 10. DMU 4_03 has an efficiency score of %. A score of 97.78% means that DMU 4_03 uses 2.22 % more units of input per output than the efficient DMU's. Table 10 shows the input and output weights as determined by the DEA problem. Banxia Frontier normalizes the weights so they sum to 100. The weight values are a percent of the total that each input or output is worth to the individual DMU. These weights maximize the efficiency of DMU 4_03 subject to the constraints of each district in State 4 not exceeding 100% efficiency. DMU 4_03 has an efficiency score of 97.78% which means there were other DMU's that reached 100% efficiency and constrained DMU 4_03 from being 100% efficient. Those DMU's, 4_12 and 4_21, are used for benchmarking. Using the dual prices from these DMU's, the target output values are calculated for DMU 4_03. Currently, DMU 4_03 has 0.94 Math, 0.83 Reading, and 0.94 Writing pass rates. The expected output would be 0.97 Math, 0.96 Reading, and 0.96 for Writing to achieve efficiency relative to the other DMU's. DMU 4_03 potentially could form a peer relationship with these DMU's 4_12 and 4_21 in order to identify areas for improving efficiency. Efficiency Score 97.78% Variables Weights Actual Target Potential Improvements Math % Reading % Writing % Free and Reduced Lunch % Expenditures/Pupil % Table 10: DMU 4_03 District-within-state (EFF W ) Peer Groups 4_12, 4_21 Benchmarking also occurs when looking at the results of district-within-all-states (Table 11). In addition to DMU's 4_12 and 4_21 being identified as peer groups for DMU 4_03, two districts from other states are added, DMU 5_05 and 5_27. Although these districts have very small dual prices which contribute to the overall target 23

28 improvement for DMU 4_03, they are still potential resources for benchmarking as the district strives to identify the causes of relative inefficiency. Efficiency Score 97.42% Variables Weights Actual Target Potential Improvements Math % Reading % Writing % Free and Reduced Lunch % Expenditures/Pupil % Table 11: DMU 4_03 District-within-all-states (EFF A ) Peer Groups 4_02, 4_12, 5_05, 5_ Decomposing Efficiency As explained in the analysis section, the "root" cause of inefficiency was determined. In order to illustrate the results, five DMU's from State 4 have been chosen: DMU 4_02 and 4_15 had EFF W = 100 and EFF A = 100; DMU 4_06 and 4_25 had had EFF W = 100 and EFF A < 100; and DMU 4_03 had EFF W < 100 and EFF A < 100. The selected DMU's in Table 12 illustrate the possible results. The average efficiencies for each state were computed. The Average EFF S for State 4 indicates that if districts are 100% efficient within their state, then % of the time they will be efficient within all states. This high percentage suggests that inefficient districts in State 4 are most likely the root of the problem. DMU EFF W EFF A EFF S "Root" cause of inefficiency 4_ N/a 4_ District 4_ State 4_ N/A 4_ State Table 12: "Root" Cause of Inefficiency 24

29 Average EFF W Average EFF A Average EFF S State State State State State Table 13: Average State Efficiencies The authors of this technique make an important point that although this method determines the "root" of inefficiency, it does not prescribe what might be causing this inefficiency. That matter is left to the DMU to determine. However, the benchmarking component of DEA, would provide peer groups that might help identify the shortfalls when a DMU is determined to be the cause of inefficiency. 25

30 Chapter 5 Recommendations and Conclusion Problems encountered while collecting data from five different states emphasized the need for a national standard for defining terms and procedures of data collection. This is especially important in regards to data that is generally accepted as correlated with educational outcome. Finding a common cross selection of data to use for this research was difficult and limited the potential inputs for the DEA problem. There are some natural extensions of this research. One extension is to consider the efficiency scores of districts as the basis for "adequate yearly progress" verses the current formula in the No Child Left Behind Act which proposes that all schools have a 100% pass rate by year Furthermore, the "root" of inefficiency analysis could be extended to national level or used to understand the effect of different types of districts such as Urban/Rural, size of enrollment, private/public, etc. Another possibility would be to conduct this same analysis in a longitudinal study in order to see trends in relative efficiency. A longitudinal study could consider the merit of DEA as a technique to be used year after year. The current ranking systems of states are popular because they are easy to implement and understand. These rankings focus only on the output of districts, and do not consider the inputs. For example, if company A produced 500 pants in a day and company B produced 100 pants in a day, focusing on outputs, company A would be rated higher than company B. However, the underlying assumption is that the inputs are the same. However, if company A uses 20 employees and 500 square feet of material a day, and company B uses 2 people and 100 square feet of material a day, focusing on inputs as well as outputs, company B would be rated higher in regards to efficiency. The same concept applies to districts. The current ranking systems assume equal inputs, and that is an incorrect assumption. Districts have different levels of resources. Therefore, DEA is a more objective and fair means of ranking districts. DEA identified districts that are efficient given the current level of their resources. Since rankings are used to identify both sanctions and rewards for districts, the importance of objective rankings is significant. According to the discrepancy between DEA efficiency scores and current rankings, there are schools who are relatively 26

31 efficient with rankings as low as 5. These schools could receive sanctions, and yet they are relatively efficient. Finally, DEA provides "peer groups" for inefficient schools so that they can evaluate their inefficiencies and improve. State Departments of Education should explore the possibility of using DEA to determine which districts are performing to the best of their abilities. This information can be used in conjunction with the pass rate ranking system to highlight the best performing districts and achieve a fairer picture of performance. 27

32 Appendix A Efficiency Scores and Peer Groups DMU EFF W Peers (EFF W ) EFF A Peers (EFF A ) EFF S 1_05, 3_23, 4_02, 1_ _ _ _02, 2_02, 3_23, 1_ _14, 5_ _ _02, 1_03, 1_ _02, 1_05, 2_02, 4_ _ _ _02, 4_12, 5_ _ _01, 1_06, 1_ _02, 1_05, 3_23, 5_ _ _03, 1_05, 1_ _02, 1_05, 4_12, 5_ _ _23, 4_12, 5_05, 5_ _ _01, 1_02, 1_05, 1_06, 1_ _02, 1_05, 4_02, 5_ _ _02, 1_ _05, 2_ _ _02, 1_03, 1_05, 1_06, 1_ _02, 1_05, 3_23, 4_02, 5_ _ _01, 1_02, 1_06, 1_ _02, 3_23, 4_12, 5_ _ _02, 1_ _05, 2_ _ _01, 1_ _05, 4_02, 5_ _ _02, 1_ _05, 3_23, 5_ _ _01, 1_02, 1_ _05, 4_12, 5_ _ _01, 1_ _05, 4_12, 5_ _ _01, 1_ _05, 4_12, 5_ _ _02, 1_ _05, 3_23, 4_02, 5_ _ _01, 1_ _05, 4_12, 5_ _ _06, 1_ _02, 4_12, 5_ _ _02, 1_ _05, 3_23, 5_ _ _02, 1_ _05, 3_23, 5_ _ _01, 1_02, 1_05, 1_ _02, 1_05, 4_12, 5_ _ _01, 1_ _05, 4_02, 5_ _ _01, 1_ _05, 4_12, 5_ _ _06, 1_ _05, 4_12, 5_