Alternative Methods to Examine Hospital Efficiency Data Envelopment Analysis and Stochastic Frontier Analysis

Transcription

1 Seminar (SEd) Nr Operations Research im Gesundheitswesen o.univ.-prof. Dr. Kurt Heidenberger Alternative Methods to Examine Hospital Efficiency Data Envelopment Analysis and Stochastic Frontier Analysis Stefan Staevski

2 Table of Contents Table of Contents Introduction Types of efficiency The methodologies CCI, 2CCI and 3 CCI Diagnosis Related Groups (DRGs) and Casemix The Casemix Indices in the UK Data Envelopment Analysis The Model Technical Efficiency with DEA Parametric techniques The United Kingdom NHS Market Case Study CCI, 2CCI and 3CCI Data Envelopment Analysis (DEA) Stochastic Frontier Analysis (SFA) Comparison of the efficiency scores Pearson Correlation Matrix Examination of the movement in efficiency scores Conclusions...30 Table of References...33 Table of Abbreviations...35 Table of Figures, Tables and Equations

3 1. Introduction Promoting public sector efficiency remains an important concern for many governments. Lacking competitive pressures, traditionally it has been held that the public sector has little inherent incentive to pursue efficient behavior [3]. In recent years, governments have sought to encourage efficiency by simulating the effects of competition. For instance, in the United Kingdom, an `internal market' was introduced in the National Health Service (NHS) in Under this arrangement, separate institutions are assigned managerial responsibility for the functions of supply and demand. The production of services becomes the sole concern of providers, such as hospitals and nursing homes. Demand for services is expressed by budget holding health authorities and general practitioners, commissioned to secure health improvements for their resident populations by buying health services. The mainstay of the internal market is the process of competitive tendering, or contracting, designed to promote competition among providers. The contractual process is supposed to force providers continually to seek to improve their productive processes, the more efficient winning more contracts by submitting bids of lower cost and/or higher quality than their competitors. However, it is now no longer widely held that the competitive pressures of the internal market are sufficient to encourage efficiency gains among hospitals, most of which enjoy local monopoly power. Instead, the competition' has been replaced by a desire to encourage cooperation in the health care sector (NHS Executive 1997). Cooperative behavior may encourage hospitals to avoid duplication of services and to secure cost reductions through exploitation of economies of scale and scope but, equally, cooperation could lead to collusion and inefficient behavior. Recognizing this possibility, the English Department of Health (DoH) has adopted a more direct mechanism to promote efficiency gains. The official method is to undertake relative performance evaluation, whereby each hospital's costs are compared to those observed in other hospitals. This policy accords with the theory of yardstick competition or of benchmarking. Indeed, the government first announced the policy as a benchmarking exercise (NHS Executive 1997). Yardstick competition or benchmarking is of particular relevance in contexts where - 3 -

4 regulators are poorly informed about the specific conditions facing each organization. Theory suggests that comparison across organizations will lead to more accurate assessment of the relationship between observed costs and effort. The methodology of the Department of Health consists [1] of comparison of NHS hospitals (Trusts) to their peers and the aim is to identify under-performing Trusts so that appropriate corrective actions might be taken. Several measures have been used to benchmark Trusts including NHS Efficiency Indices, NHS Performance Indicators, the Labor Productivity Index (LPI) and cost information on NHS Trusts in the form of league tables. More recently, the Department of Health has proposed regression analysis be applied to Trust data to develop three cost indices (the CCI, 2CCI and 3CCI) which can be used to produce productivity rankings. Our study uses the same Department of Health dataset of NHS Trusts and employs two different approaches that have been more generally used to study hospital efficiency. The purpose of the study is to then compare the efficiency rankings of the three cost indices (CCI, 2CCI and 3CCI) with those from other methodologies (Data Envelopment Analysis (DEA) and Stochastic Frontier Analysis SFA)) to assess whether these alternative techniques can explain anything over and above the conventional regression approach and whether there is any inconsistency between the efficiency estimates produced. In order to answer the research question, the efficiency rankings of NHS Trusts from the three cost indices CCI, 2CCI and 3CCI are compared with rankings produced using DEA and SFA methodologies. The same variables are used so as to make the three methods as comparable as possible and then evaluate the efficiency scores that each method produces. For the DEA and SFA techniques, five (identical) model specifications are tested which match as far as possible 2CCI and 3CCI specifications as well as others to enable sensitivity analysis

5 2. Types of efficiency At first, it is useful to look at a number of 'types' of efficiency and the way in which they relate to each other. Consider the concept of economic efficiency, which is composed of technical and allocative efficiency. Diagram 1- Types of Economic Efficiency Nunamaker (1985) [4] defines technical efficiency as a measure of the ability of a micro level unit (referred to as a firm, observation or decision making unit (DMU) to avoid waste by producing as much output as input usage will allow, or using as little input as output level will allow. Allocative efficiency measures the ability of a DMU to avoid waste by producing a level of output at the minimal possible cost. Another decomposition occurs at the level of technical efficiency, which can be considered to be composed of scale and non-scale effects, the latter being referred to as pure technical efficiency. Scale efficiency is the measure of the ability to avoid waste by operating at, or near, to the most productive scale

6 Lastly, pure technical efficiency can be considered to be composed of congestion efficiency and other effects. Input congestion efficiency is the measure of the component of pure technical efficiency due to the existence of negative marginal returns to input, and the inability of a firm to dispose of unwanted inputs costlessly. 3. The methodologies After explaining the different types of efficiency, we will proceed with a brief overview of the methodologies that we use CCI, 2CCI and 3 CCI Diagnosis Related Groups (DRGs) and Casemix The Diagnosis Related Groups (DRGs) are a patient classification scheme which was originally developed as a means of relating the type of patients a hospital treats (i.e., its casemix) to the costs incurred by hospitals. The design and development of the DRGs began in the late sixties at Yale University (Fetter, et al, 1980). The initial motivation for developing the DRGs was to create an effective framework for monitoring the utilization of services in a hospital setting. [5] The original purpose of the DRGs was to relate the casemix of a hospital to the resource demands and associated costs experienced by the hospital. Thus, DRGs focused exclusively on resource intensity. Therefore, a hospital having a more complex casemix from a DRG perspective meant that the hospital treated patients who required more hospital resources, but not necessarily that the hospital treated patients having a greater severity of illness, a greater risk of dying, a greater treatment difficulty, poorer prognoses or a greater need for intervention. As the health care - 6 -

7 industry has evolved there has been increased demand for a patient classification system that can be used for applications beyond resource use, cost and payment. In particular, a patient classification system is needed to o Compare hospitals across a wide range of resource and outcome measures o Evaluate differences in inpatient mortality rates o Implement and support critical pathways o Facilitate continuous quality improvement projects o Support internal management and planning systems o Manage capitated payment arrangements The Casemix Indices in the UK The National Health System (NHS) Cost Indices [6] published in are RCI, RCI+, CCI, 2CCI and 3CCI. The National Reference Cost Index (RCI) is compiled using data provided by hospitals about their unit costs by Healthcare Resource Group (HRG) for their main surgical specialties. The RCI is a weighted average of all HRG costs in each hospital relative to the national average. The Market Forces Factor (MFF) is then added to account for differences in local factor costs. The index was published in November The RCI covers only acute activity. The RCI+ attempts to provide a comprehensive indication of activity, by including non-surgical, outpatient and accident and emergency (A & E) activity. The Casemix Cost Index (CCI), unlike the RCI, includes mental health services and day care costs. The index is a ratio of actual to expected costs, taking into account hospital casemix. Activity in the CCI is summarized as a weighted combination of HRG based inpatient spells, outpatient first attendances and A & E first attendances

8 The Casemix Costliness Cost Index (2CCI) builds on the CCI, incorporating adjustment for other variables hypothesized to explain cost differences among hospitals. These variables included hospital transfers, multi-episode spells, and the proportion of elderly or female patients, student numbers, research revenue and the MFF. The extent of the adjustment for each of these variables is estimated through regression analysis. The 3CCI the Casemix Costliness and Configuration Cost Index attempts to take into account differences in hospital configuration, over and above the adjustments made in the 2CCI. These include the costs of multi-site working, hospital size, and capacity utilization Data Envelopment Analysis While there is general agreement about the applicability of Data Envelopment Analysis (DEA) to evaluate hospital efficiency, a number of features of the model may worry many researchers in the field. Two important problem areas of the model are: the assumption that there is no 'noise' (or error) in the data being studied; and the lack of a definite functional form encapsulating the production technology. The latter, whilst a strong argument for the technique in many studies, raises the problem of what method should be used to evaluate the results of a DEA study, mainly due to the inability to perform the usual diagnostic tests associated with regression estimation. As the earlier list of applications suggests [8], DEA can be a powerful tool when used wisely. A few of the characteristics that make it powerful are: o o o o DEA can handle multiple input and multiple output models. It doesn't require an assumption of a functional form relating inputs to outputs. DMUs are directly compared against a peer or combination of peers. Inputs and outputs can have very different units. For example, X1 could be in units of lives saved and X2 could be in units of euros ( ) without requiring an a priori tradeoff between the two

9 The same characteristics that make DEA a powerful tool can also create problems. o o o o Since DEA is an extreme point technique, noise (even symmetrical noise with zero mean) such as measurement error can cause significant problems. DEA is good at estimating relative efficiency of a DMU but it converges very slowly to absolute efficiency. In other words, it can tell you how well you are doing compared to your peers but not compared to a theoretical maximum. Since DEA is a nonparametric technique, statistical hypothesis tests are difficult and are the focus of ongoing research. Since a standard formulation of DEA creates a separate linear program for each DMU, large problems can be computationally intensive The Model The original Charnes, Cooper and Rhodes (1978) [7][8] paper considered an input-oriented, constant returns to scale (CRS) specification, with additional modifications to the methodology including a variable returns to scale (VRS) model and an output-oriented model. The Charnes, Cooper and Rhodes (1978) paper reformulated Farrell's original ideas into a mathematical programming problem, allowing the calculation of an efficiency 'score' for each observation in the sample. This score is defined as the percentage reduction in the use of all inputs that can be achieved to make an observation comparable with the best, similar observation(s) in the sample with no reduction in the amount of output. Equation (1) [9] below sets out the linear programming problem corresponding to the basic DEA specification of Charnes, Cooper and Rhodes (1978). This linear program (LP) is in fact the dual, envelopment form of an efficiency maximization LP for each observation. The objective function seeks to minimize the efficiency score, θ, which represents the amount of radial reduction in the use of each input. The constraints on this minimization apply to the comparable use of outputs and inputs. Firstly, the output constraint implies that the production of the rth output by observation i cannot exceed any linear combination of output r by all firms in the sample. The - 9 -

10 second constraint involves the use of input s by observation i, and implies that the radially reduced use of input s by firm i (θ) cannot be less than the same linear combination of the x is use of input s by all firms in the sample. In other words, to reduce the use of all inputs by observation i to the point where input usage lies on the 'frontier' defined by the linear combination of input and output usage by the 'best' firms in the sample. Considering a dataset containing K inputs, M outputs and N firms, where the sets of inputs and outputs for the ith observation are x ik, k=1,...,k and y im, m=1,...,m, the input-oriented CRS DEA LP for observation i has the form: Equation 1, [9] where θ is a scalar and λ is an N x 1 vector of constants. The value of θ obtained from the LP is the efficiency score for the ith observation, and will lie in the region (0,1]. An efficiency score of 1 indicates a point on the frontier and hence a technically efficient observation relative to the dataset Technical Efficiency with DEA Equation (1) must be solved N times, once for each observation in the sample. The efficiency scores from the set of LPs (1) indicate, given a level of output, by how much inputs can be

11 decreased for an inefficient observation to be comparable with similar, but more efficient, members of the sample. This efficiency is often referred to as technical efficiency. Figure 1- DEA - Example As an illustration of the technique, consider an example of six firms using two inputs (input 1 and input 2) to produce one unit of output, shown in Figure 1. The linear programming solution produces the non-parametric piece-wise linear frontier (ss'). Firms which lie on this frontier are fully efficient (firms a, b, d, f). Firms which lie above and to the right of the frontier are inefficient (firms c and e). The measure of the technical inefficiency of firm c (θ from equation 1) is captured by the ratio 0c'/0c. Note that point c' in Figure 1 does not represent a firm, but the point on the frontier that firm c would occupy if it could be made fully efficient by radially reducing its use of both inputs. That is, firm c could reduce the amount of input 1 and input 2 it uses in production and still produce the same amount of output. The input-oriented DEA technique calculates efficiency scores by the amount of radial reduction in inputs that can be achieved to move the firm towards the best practice frontier. By using the radial reduction technique (moving each inefficient firm towards the frontier by contracting towards the origin each input by the same proportion), the technique becomes invariant to the units used to measure each input

12 Equation (1) represents the case in which the assumption of constant returns to scale is imposed on every observation in the sample. In this formulation no account is taken of factors which may make firms unique beyond the simple input-output mix, such as inefficiencies which result from operating in areas of increasing or decreasing returns to scale due to size constraints. Another assumption embodied in the LP in equation (1) is that of strong disposability of inputs. This represents the assumption that, when reducing input usage, an observation is able to dispose of the unwanted inputs costlessly. In effect, this assumption rules out the possibility of decreasing marginal products for inputs. To further decompose the efficiency scores from equation (1) it is necessary to use a number of additional DEA formulations which relax some or all of the assumptions embodied in the basic DEA equation. The first variation relaxes the CRS assumption by considering scale and allowing firms to exhibit both increasing and decreasing returns to scale in addition to constant returns. Known as the VRS formulation, this involves the addition of a constraint to the basic CRS formulation specifying that the sum of the linear combination parameters be equal to one ( Σj λj = 1 ). In practice, this most often results in a 'tighter' fitting frontier with more firms on and near to the frontier (efficiency scores closer to one). The efficiency scores from models estimating CRS and VRS, can be used to calculate scale efficiency or each observation (θ SC ) using the following relationship between CRS (technical efficiency) und VRS (pure technical efficiency) efficiency scores: θ CRS = θ VRS.θ SC (Equation 2)

13 3.3. Parametric techniques Recently a number of studies have applied SFA to hospital datasets [3], usually to measure relative efficiencies. In all these studies cost functions were estimated rather than production functions. Cost functions are often estimated to control for the biases that arise in the direct estimation of production functions, so as to represent a multi-product firm, or because analysts are interested in measuring allocative efficiency as well as technical efficiency or a combination of these two efficiencies, cost efficiency. Stochastic frontier modeling is becoming increasingly popular primarily because of its flexibility and its ability to closely marry economic concepts with modeling reality. These techniques are also now more easily applied given improvements in computing technology and the availability of unit record datasets. Stochastic frontier modeling is often used to compare firms' relative efficiencies though it can also be used to derive estimates of productivity change over time. The technique has a number of benefits when compared to standard econometric estimation (OLS) of production functions. It estimates a 'true' production frontier rather than an average frontier, thus it fully represents the maximal properties of the production function. One important implication of estimating the frontier is that measured productivity change will represent pure technological change rather than a combination of efficiency change and technological change which is the case when using non-frontier techniques. However, OLS estimation of functions is still very useful when testing for standard statistical aspects of the analysis, for example, heteroskedasticity and the normality of the residuals. SFA is a statistical technique [2] that generates a stochastic error term and an inefficiency term by using the residuals from an estimated production or cost frontier. The stochastic cost frontier model is typically defined to be:

14 Equation 3, [1]-p.105 vi is caused by stochastic disturbances, for example, unexpected expenditures for hospital repairs, unexpected winter pressure on beds arising from cold weather or a temporary local outbreak of disease, as well as measurement error in the cost variable and omitted explanatory variables. The ui is the degree of inefficiency or the distance from the cost frontier. Although the two components of the residual can have a number of different distributions, a common assumption in the estimation procedure is that vi is normally distributed, while ui is often represented by a half-normal distribution. If allocative efficiency is assumed, the ui is closely related to the cost of technical inefficiency. Technical inefficiency may arise from managerial slack (X-inefficiency), outmoded equipment, or inadequate staffing. If the assumption of allocative efficiency is not made, the interpretation of the ui in a cost function is less clear, with both technical and allocative inefficiencies possibly involved. Predictions of individual firm cost efficiencies are defined [1]

15 EFF i will take a value between one and infinity and can be [1] These scores are then inverted to define efficiency as 0 < 1/EFF i < 1. SFA has some advantages over non-parametric techniques, such as DEA, for estimating frontiers, efficiency and productivity; in particular, it is able to account for measurement error. However, stochastic frontier modeling does have some constraints which DEA does not including: only one output can be accommodated when modeling production functions, and the need to select functional forms for both the production structure and error components. Thus, parametric techniques for measuring efficiency and productivity provide an alternative approach for dealing with errors, at the cost of using a more restrictive model specification than DEA. Parametric techniques such as SFA are also more likely to be appropriate if the focus is on drawing conclusions about the aggregate properties of the dataset, rather than the performance of individual units. By contrast, DEA may be more appropriate if the focus is on developing a detailed understanding of the performance of individual units within the sector or identifying DEA peer relationships among the production units

16 4. The United Kingdom NHS Market Case Study 4.1. CCI, 2CCI and 3CCI As we already know, three separate cost indices have been developed by the Department of Health to produce efficiency rankings for Trusts in order to benchmark their performance based on their productivity scores. This analysis was based on 1995/6 data using a deterministic OLS. The CCI cost index is a measure of actual divided by expected costs, where expected costs are average national costs per respective attendance and activity measures include case-mix adjusted inpatient, first outpatient and accident and emergency (A & E) attendances. The CCI was developed so as to be plausible, intuitive and understandable to health service managers, yet consistent and robust in its identification of poorly performing Trusts. As such it includes a number of deterministic adjustments in contrast to the conventional econometric approach of including all adjustments on the RHS of the equation. The derivation of the CCI is as follows [1]

17 Equation (4), [1]- p.114 2CCI and 3CCI are long and short run indices regressed against the CCI with increasing numbers of explanatory variables. 2CCI takes factors into account such as additional adjustments for case mix, age and gender mix, transfers in and out of the hospital, inter-specialty transfers, local labour and capital prices and teaching and research costs for which Trusts might be over or under compensated. The 3CCI makes additional adjustments over and above those in the 2CCI for hospital capacity, including number of beds, and number of sites, scale of inpatient and non-inpatient activity and scope of activity. It therefore tries to capture institutional characteristics amenable to change in the long, but not the short run. The results from the regression are

18 Table 1, [1]-p.107 The regressions were run on a sample of 232 Trusts and productivity scores were then derived from the residuals of the regressions. In order to compare the three cost indices with DEA and SFA models, the efficiency scores were re-scaled so that the most efficient Trust would rank as one and inefficient Trusts as less than one. The basic descriptive statistics for these efficiency indices are shown in the table below Table 2, [1]-p

19 4.2. Data Envelopment Analysis (DEA) DEA model specifications were developed based on the same variable set as the above cost index regression models, in order to produce efficiency scores that would be most comparable. The DEA methodology uses the relationship between inputs and outputs to establish efficiency scores. The input used in the DEA analysis in each case, is the cost index CCI, the dependent variable from the regression model. The cost index as the input is unusual as it already represents a casemix adjusted efficiency measure, but it was kept as the input to maintain as much consistency as possible with the original regression thus enabling greater comparability across the different methods. Using the CCI is therefore a simple extension of the unit cost analysis of the Department of Health with variables on the left and right hand side simply divided through by activity. Using the CCI and not total cost as the input was also necessary to ensure that all variables were in ratio form. Only ratio or index variables were used (for example, transfers per spell, episodes per spell) and so scale of activity variables (total number of spells, beds, total outpatient and total A& E attendances) from the regression model were excluded. The only non-ratio variable left in the model was SITES50B (for detail see table 3 on the next page), sites with more than 50 beds, which effectively means that Trusts are compared to other Trusts in the analysis with the same number of sites as themselves or with others on more sites. With the cost index as the input, various specifications were attempted using the selection of variables from the regression model listed in the table below. All outputs for the DEA were transformed so as to be positively related to efficiency, for example, transfers out per spell became (1-(transfers out per spell))

20 Table 3, [1]-p.107 Five model specifications (for details see table 4 on the next page) were employed using the above listed variables. Although any number and combination of variables could have been included as outputs, the following specifications were used in order to maintain some theoretical grounding and reasoning for their inclusion. Specification 1 uses all the outputs listed above and all the other specifications are therefore nested within the first one. Specification 2 uses only the variables from the benchmarking regression to obtain the short-run efficiency index 3CCI, the scope and capacity variables. The third specification uses the variables that were highly significant in the cost index regression models (both the full model and the trimmed model with the outliers excluded). Specification 4 includes those variables for which there was some a priori hypothesis that they were positively correlated with cost and specification 5 uses the variables that were significant in the full regression model (as shown in table 1). In each case, each Trust s performance is assessed on those outputs that are included. The 5 specifications and the results are shown in the table below:

21 Table 4, [1]-p.109 Given the nature of the data (ratio/proportional), a variable returns to scale (VRS) model was run, which because of the use of ratios, effectively implies a constant returns to scale (CRS) model

22 As seen in table 4, the full set of variables used in specification 1 produced the higher efficiency scores with a mean of 0.94 and a lower standard deviation than the other specifications. In general, as the other specifications were nested within number 1, the efficiency scores increased as more variables were added. A higher number of Trusts also fell on the efficiency frontier in specification 1 than in any of the other specifications, which is to be expected. As more variables are added, efficiency scores increase, variability decreases and a greater number of Trusts end up on the efficient frontier with scores of 1, thus rendering the specification less discriminating. The different specifications also serve in some sense as a sensitivity analysis, as the individual scores remain relatively stable when parameters are removed and then added again. Because one has no diagnostic tools with which to choose the best model specification, some general rules of thumb apply. The most important criterion for selecting one of these specifications is whether the model is consistent with theory and in some way theoretically justifiable. Another useful criterion is the number of efficient units. Ceteris paribus, the fewer the better, although there should be enough peers available to make useful comparisons. The distribution of efficiency scores makes for another useful criterion. The wider the better, ceteris paribus. For these reasons specification 5 was selected as a good model since all the outputs in this specification were highly significant variables in the original full regression model. Figure 2 shows the frequency distribution of efficiency scores for the 5 DEA specifications and highlights that specification 1 produces the higher efficiency scores while specification 5 produces a spread of efficiency scores that are more average

23 Figure 2, [1]-p Stochastic Frontier Analysis (SFA) The regression-based technique SFA was used to validate and compare the efficiency rankings of the 5 DEA specifications. The five specifications were exactly replicated with the cost index (input) once again as the dependent variable and the outputs specified in each, as the regressors. A half-normal distribution was assumed for the error term and a linear functional form so as to be most comparable to the original linear Department of Health regression model. Table 5 gives the basic descriptive statistics for the efficiency scores of the 5 specifications

24 Table 5, [1]-p.110 Figure 3 displays the distribution of the efficiency scores from the 5 SFA specifications. Figure 3, [1]-p

25 Figure 3 shows a much narrower distribution of scores in general compared to DEA with relatively little variability across the 5 specifications. Stochastic specifications 1 and 5 also produce extremely similar distributions, unlike DEA specifications 1 and Comparison of the efficiency scores Pearson Correlation Matrix At first we can compare the distribution of efficiency for the CCI, DEA and SFA methods (for better overview here once again Figure 2 and Figure 3): Figure 4, [1]-p

26 Figure 2, [1]-p.109 Figure 4, [1]-p

27 All three cost indices appear to have more skewed distributions than the 5 DEA specifications shown in figure 2 and as expected, the three cost indices have slightly lower average efficiency scores than the 5 SFA specifications in figure 3 since OLS is deterministic and does not separate inefficiency from random noise. There also does not appear to be as much variability between them as between the 5 DEA distributions. Table 6 shows the correlations between the cost indices and the efficiency scores from the 5 DEA and SFA specifications. Table 6, [1]-p.111 There is a high degree of correlation (around 0.7) between the two regression based techniques (the OLS regression and SFA). Correlations within the 5 SFA specifications are also igh, around 0.8, and within the 3 cost indices, around 0.7. While they are also quite high within the 5 DEA specifications, it is generally more variable. However, the correlations between the 5 DEA specifications and the other two techniques are generally lower and more disappointing. In specifications 2, 3 and 5 reasonable correlations are achieved with the cost indices of around 0.5 and all others are positive. It is also worth noting that correlations for the DEA tend to improve across the three CCIs as more explanatory variables are added. The correlations between the two methods DEA and SFA within certain specifications (along the diagonal) are also relatively high, especially for

28 specifications 2, 3 and 5 again which are between 0.59 and Within the same specifications there may therefore be more agreement, but across different specifications and methods, the correlations fall. There do, however, appear to be some major anomalies for individual Trusts and the techniques do not appear to be measuring efficiency related to cost in entirely the same way. The relationship does appear to be sensitive to how the models are specified Examination of the movement in efficiency scores Correlations are also not an entirely satisfactory way to examine the changes in efficiency scores across different methods and specifications, as they do not show what happens to individual Trusts scores

29 Table 7, [1]-p.112 Table 7 examines the movement in efficiency scores between a number of the specifications used in the different methods in relation to SFA specification 5 when grouped into deciles. SFA-5 was chosen because DEA specification 5 was selected as a good model (in terms of its spread of efficiency scores and being based on significant variables from the original regression). SFA specification 5 is therefore compared to the cost index, to SFA specification 2 and to DEA specification 5. These had correlations with SFA specification 5 of 0.58, 0.65 and 0.63, respectively. The efficiency scores are grouped in deciles with respect to SFA specification 5 and all scores on the diagonal would represent a perfect correlation

30 The results in table 7 show that mostly SFA-5 has higher scores than the others and as such most scores fall to the left and below the diagonal. In particular, for the cost index CCI, listed first, no scores lie to the right of, or indeed on the diagonal. In fact all scores lie more than one decile away and are quite disparate from the stochastic frontier efficiency scores. For SFA specification 2, only 5% of the scores lie more than a decile apart, thus there is a high degree of congruence between them. There is a relatively high degree of agreement between DEA specification 5 and SFA specification 5 and only 12% of them fall more than 1 decile away from each other. Thus although the correlation of DEA-5 with SFA-5 (0.63) would appear only marginally higher than that for SFA-5 with the cost index (CCI) (0.58), there is clearly a higher consistency between the former results as shown by the degree to which the scores shift across deciles. This would seem to suggest that although the different methods, for example DEA and SFA, appear to be somewhat inconsistent, they do have congruity within specifications and may complement one another. This also suggests that correlation as a method may not be adequate to capture the subtler shifts in efficiency for individual Trusts across different methods. 5. Conclusions In our study we focused on the consistency and robustness of efficiency scores across the DEA and SFA techniques when applied to the same dataset. Sensitivity analysis was carried out within the DEA and SFA models by changing the model specifications including and excluding different variables, and testing for the robustness of the results. While the models proved to be robust in this respect, there was some inconsistency across the different methodologies

31 Caution is therefore warranted against literal interpretations of Trust efficiency scores and rankings. Reasonable correlations (see 4.4.1) might have suggested convergent validity but these were at best modest across the different techniques. Reasons that have been proposed for the lack of agreement include the way outliers or extreme data points are treated in the different methods and the fact that correlations may not necessarily be the best way to examine the relationship between sets of scores for individual Trusts. Another possible reason for the lack of agreement across the different methods is that there appears to be a large amount of random noise in the study which could potentially be mistaken for inefficiency. If the DEA and SFA analyses used the same set of variables and there was no random noise, then they would give extremely consistent results. However, as there are differences between the efficiency scores, even in the specifications using the same sets of inputs and outputs, it is either due to misspecification in the models or high levels of random noise. Another possible reason for the lack of congruence is that DEA addresses the issue of technical efficiency whereas the inefficiency measured by SFA may be a combination of technical and allocative inefficiency and without further assumptions the SFA method may be unable to separate the two sources (see 3.3.). The distinction between allocative and technical efficiency is important, since each would require a different policy response. The different efficiency scores should not therefore be interpreted as accurate point estimates of efficiency, but might more usefully be interpreted as indicating general trends in inefficiency for certain Trusts. The point estimates of inefficiency in either method are indeed sensitive to specification, measurement and data errors. However, when several specifications were used, general trends could be discerned as to which Trusts usually came out as being more efficient and which ones generally emerged as inefficient. It is therefore imperative that several specifications be employed to gauge an overall picture of efficiency. Given cross-sectional data, these techniques are certainly some of the better methodological approaches available and where possible should be used in conjunction with one another as the two techniques are complementary in many respects. Both methods serve as signaling devices. To the extent that there is no a priori reason to prefer one methodology over another and as long as there is no solution to the problem of choosing the best reference technology (and there simply may be no

32 solution), it seems prudent to analyze efficiency using a broad variety of methods to crosscheck. Ultimately, data accuracy is paramount to any such analysis as inaccurate data in, for instance the DEA methodology, will affect not only that Trust s efficiency rating but also potentially the efficiency ratings of other Trusts as well. The level of random noise is a reflection of the quality of the data and will affect the ability to measure efficiency. From the Department s point of view, improving on significant data deficiencies would probably contribute more to better efficiency estimates than further experimentation with alternate specifications and estimation techniques. In particular, data on research and teaching outputs and quality of care and outcome indicators would be important. There may not be a strong self-interest in the accurate reporting of data and as such incentives might be needed to ensure this. Future research should also consider ways to improve models through the possible inclusion of some alternative variables

33 Table of References [1] Rowena Jacobs, Alternative Methods to Examine Hospital Efficiency: Data Envelopment Analysis and Stochastic Frontier Analysis, Centre for Health Economics, University of York, Heslington, York Y010 5DD, UK [2] T. Coelli, A guide to FRONTIER version 4.1: A computer program for stochastic frontier production and cost function estimation, Centre for Efficiency and Productivity Analysis, CEPAWorking Paper 96/07, University of New England (1996) - p.8 [3] Andew Street and Rowena Jacobs, Relative Performance Evaluation of the English acute hospital sector, Centre for Health Economics, University of York, Heslington, York Y010 5DD, UK ( , [4] Nunamaker, Thomas R. 1985, Using data envelopment analysis to measure the efficiency of non-profit organisations: A critical evaluation, Journal of Managerial and Decision Economics, vol. 6, pp ( , [5] The Evolution of Casemix Measurement Using Diagnosis Related Groups (DRGs), Richard F. Averill, M.S., John H. Muldoon, M.H.A., James C. Vertrees, Ph.D., Norbert I. Goldfield, M.D, Robert L. Mullin, M.D., Elizabeth C. Fineran, M.S., Mona Z. Zhang, B.S., Barbara Steinbeck, ART, Thelma Grant, RRA ( , [6] Andew Street, Confident about efficiency measurement in the NHS?, Centre for Health Economics, University of York, Heslington, York Y010 5DD, UK ( , [7] DEA Homepage of University of KENT, UK ( ,

34 [8] Portland State University, USA, DEA Homepage ( , [9] Richard Webster, Steven Kennedy, Leanne Johnson, Comparing Techniques for Measuring the Efficiency and Productivity of Australian Private Hospitals ( , cdb3dbbf85ca25671b001f1910/$file/13510_nov98.pdf) pp

35 Table of Abbreviations ABBREVIATION MEANING 2CCI Casemix Costliness Cost Index 3CCI Casemix Costliness and Configuration Cost Index A & E Activity Accident and Emergency Activity CCI Casemix Cost Index CRS Constant Returns to Scale DEA Data Envelopment Analysis DMU Decision Making Unit DoH English Department of Health DRG Diagnosis Related Groups HRG Healthcare Resource Group LP Linear Program LPI The Labor Productivity Index???? MFF Market Forces Factor NHS National Health System RCI, RCI+ The National Reference Cost Index SFA Stochastic Frontier Analysis VRS Variable Returns to Scale

36 Table of Figures, Tables and Equations FIGURE/EQUATION DESCRIPTION Diagram 1 Types of Economic Efficiency Equation 1 the input-oriented CRS DEA LP for observation i Equation 2 relationship between CRS (technical efficiency) und VRS (pure technical efficiency) efficiency scores Equation 3 stochastic cost frontier model definition Equation 4 The derivation of the CCI Figure 1 DEA- Example Figure 2 Distribution of Efficiency scores for 5 DEA specifications Figure 3 Distribution of Efficiency scores for 5 SFA specifications Figure 4 Distribution of Efficiency scores for cost indices from regression model Table 1 Regression Results and Descriptive Statistics Table 2 Efficiency Scores from Regression Model Table 3 DEA model variables used in regression model Table 4 DEA model specifications and efficiency scores Table 5 Descriptive Statistics for SFA model efficiency scores Table 6 Pearson correlation matrix for DEA, SFA and CCI Table 7 Movement of efficiency scores