25 NOVEMBER Assessing the Value for Money of Regulation of Infrastructure Networks

Transcription

1 FOURTH ANNUAL CONFERENCE ON COMPETITION AND REGULATION IN NETWORK INDUSTRIES 25 NOVEMBER 2011 RESIDENCE PALACE, BRUSSELS, BELGIUM Assessing the Value for Money of Regulation of Infrastructure Networks Alex Scharaschkin Director, Consumers and Competition VFM, National Audit Office, Buckingham Palace Road, London, SW1W 9SP, United Kingdom +44 (0) / alex.scharaschkin@nao.gsi.gov.uk Abstract Performance measurement plays a critical role in helping organisations to manage themselves, monitoring progress against their objectives and identifying areas that require attention. In the public sector it is also used for accountability, so that taxpayers can see what has been achieved through public funding. Challenges in measuring the performance of a regulator include: determining indicators of outcomes reflecting the statutory duties set for it by Parliament; appropriately recognising the fact that outcomes are usually influenced by other factors as well as the regulator's actions; and determining thresholds for good performance, given this problem of attribution. The UK National Audit Office (NAO) holds economic regulators to account to Parliament. This paper draws together the NAO s experience of how regulators have addressed some of the challenges outlined above and the wider conclusions drawn from its Value for Money (VFM) work programme. Keywords audit; outcomes; performance indicators; performance measurement; public sector; regulation; UK; value for money Last saved 23/11/11 15:00 1

2 Contents Contents... 2 Summary... 3 Part 1 Performance Measurement Principles... 4 Part 2 The UK Regulatory Regime... 5 Case Example: trade-offs between efficiency incentives and regulatory information requirements Case example: Performance of the regulator vs. performance of the market... 6 Case example: determining and presenting relevant outcome data... 7 Case example: attributing market outcomes to regulatory interventions... 8 Part 3 Assessing the value-for-money of a regulator Conclusion Last saved 23/11/11 15:00 2

3 Summary 1. This paper summarises some of the key themes that have arisen in the recent work of the UK National Audit Office (NAO) on the performance measurement practices of economic regulators in the UK. It aims to: - provide an overview of performance measurement principles, as applied to public sector entities; - identify some of the common challenges that the economic regulators face, using illustrative case examples; and - show how it is necessary for regulators to grapple with these challenges, in order to demonstrate that they are delivering value for money. 2. The economic regulators in the UK were set up to regulate the prices charged to customers, and the markets established, following Government privatisations of the former state owned utility monopolies in the energy, water, telecommunications and transport sectors during the 1980s and 1990s. These regulators are generally funded through fees or levies on the regulated companies, and consequently their costs are ultimately passed through to customers in their bills. 3. The foremost responsibility of each regulator is to deliver its statutory objectives. These vary between regulators, and may include, for example, protecting (current or future) consumers, protecting the environment, or health and safety regulation. In working to deliver these statutory objectives, regulators must match their own finite resources against their assessment of the risks whilst also minimising the burdens and costs they place on businesses. 4. Regulators face particular challenges in trying to measure their performance, particularly in relation to: - determining indicators of outcomes that reflect the statutory duties set for them by Parliament; - appropriately recognising the fact that outcomes are usually influenced by other factors as well as the regulator's actions; and - determining thresholds for good performance, given this problem of attribution. 5. The UK National Audit Office (NAO), on behalf of taxpayers and service users, reports to Parliament on the value-for-money achieved by individual regulators. In doing so, we have observed how regulators have attempted to address some of these challenges. Last saved 23/11/11 15:00 3

4 Part 1 Performance Measurement Principles 6. Guidance published by the UK Treasury, the National Audit Office and others 1 states that a performance information system should have clear links between the high-level objectives of the organisation and its measures of success. In the case of the UK regulators, the statutory duties are general in nature and open to wide interpretation. The regulators use their discretion to translate these into strategic objectives and ultimately operational objectives within their own organisation. 7. It can help to break down an organisation s activity into inputs, outputs, and outcomes. It is important not just to measure outcomes, which tend to be more difficult to measure than inputs and outputs, but also to understand the links between inputs, outputs and outcomes in order to assess how performance can be improved. At present most regulators performance measures tend to focus on their own outputs rather than outcomes (Figure 1). Figure 1: Value for Money External factors Resources ( ) Inputs Outputs Outcomes Economy Efficiency Effectiveness Cost effectiveness / value for money Source: NAO and FABRIC 8. It is also important to develop criteria against which results can be compared, to allow an assessment of whether objectives are being met. These can take the form of targets, or softer criteria where the organisation may identify a range of acceptable outcomes, outside of which re-evaluation and intervention may be required. 1 Choosing the right FABRIC: a framework for performance information, HM Treasury, Cabinet Office, National Audit Office, Audit Commission and Office for National Statistics, 2001 Last saved 23/11/11 15:00 4

5 Part 2 The UK Regulatory Regime 9. Regulation has been used by governments for millennia, for example to enforce standard weights and measures, and to protect the currency. It is also a key tool for achieving many contemporary policy objectives, such as to drive up quality in schools; enforce professional standards of skill and training in healthcare; and protect people held against their will, in prisons and mental institutions, from abuse and mistreatment. 10. In the UK, economic regulation is largely associated with post-privatisation control over the utilities (telecommunications, gas, electricity, water and railways). However, regulation of markets already filters through policy delivery in many parts of government. It will become even more important if, as planned, the current Government transfers more powers to localities and communities, increases users choice between diverse and competing service providers and, under fiscal pressure, seeks alternatives to public provision as a way of achieving policy objectives 2. Case Example: trade-offs between efficiency incentives and regulatory information requirements. 11. Our report on the UK s Office of Rail Regulation 3 examined how effectively the regulator was performing against its statutory objective to ensure that Network Rail, the monopoly owner of the UK rail network, is operating, maintaining and renewing the network efficiently 4. To do this, we focused on the regulator s effectiveness against two key requirements: - there must be strong incentives on the regulated company to achieve efficient and sustainable levels of cost; and - there must be robust information for the regulator to judge what level of cost is efficient and sustainable, and how the regulated company s performance compares with that efficient cost. 12. Figure 2 illustrates the trade-off between market incentives and the regulator s information requirements. We consider that regulated companies are more likely to seek greater efficiency and reveal their true efficiency potential when they have strong incentives to do so from shareholders, lenders and customers. Where these incentives are weak, the regulator must rely more on good quality information on the companies costs and their potential for improvement. 2 The NAO report Oversight of user choice and provider competition in care markets, NAO, September 2011, examines the effectiveness of government regulation of a quasi-market (i.e. one in which service users are allocated public funds to purchase services from a range of competing providers). 3 Office of Rail Regulation: Regulating Network Rail s efficiency, NAO, April 2011 ( 4 Of course, the regulator also has a statutory objective to regulate the safety of the network. Last saved 23/11/11 15:00 5

6 Figure 2 Requirements for effective regulation of cost efficiency Incentives / information trade-off More efficiency incentives for companies, less information needed by regulators Market structure Fully competitive markets or degrees of contestability UK examples: Energy supply, telecommunications networks Several fully integrated geographical companies Water National networks with equity funding Electricity and gas transmission networks Fewer efficiency incentives for companies; more information needed by regulator Single network provider; debt rather than equity funding; government guarantee Network Rail Source: National Audit Office Case example: performance of the regulator vs. performance of the market 13. In common with most regulators, the UK s energy regulator, Ofgem, can only achieve most of its statutory duties and objectives through its influence on the companies that it regulates. Its performance ultimately depends on a number of factors, such as companies behaviour in relation to other industry participants (particularly consumers), and companies performance in achieving outcomes consistent with Ofgem s objectives (for example sustainable development). There is a range of tools available to the regulator to achieve these objectives: relatively informal action, such as guidance to companies; financial incentives and sanctions (through price control reviews); and more formal action under companies licences (such as enforcement orders) and statute (for example fines). Ofgem can use these tools in combination or as a regulatory escalator, relying for example on informal action or comparative competition unless it judges that more formal action is justified. 14. Regulators are presented with challenges in measuring the performance they achieve through this influence and range of tools. In Ofgem s case: - Ofgem has been given statutory duties by Parliament, but these do not have associated targets or indicators against which to evaluate performance. - Annual performance measures, particularly in infrastructure industries, can be strongly influenced by exceptional external events, and can mask underlying problems which may only become apparent in the longer term. For these reasons it is common for regulators to use a basket of indicators to judge companies performance, rather than rely on any single indicator, and to analyse trends in data not just year-on-year performance. Last saved 23/11/11 15:00 6

7 15. Ofgem has translated its statutory objectives, duties and functions into a number of themes and associated deliverables against which it measures its own performance. Our report 5 found that its performance reporting does, however, focus on its own outputs rather than outcomes, and we recommended that it should review the way it reports its performance and the scope for giving a clearer indication of the outcomes it is seeking to achieve, how these relate to its statutory duties and its performance over time in achieving these. Case example: determining and presenting relevant outcome data 16. Figure 3 shows how another regulator draws together information across a wide range of regulated company activities. The UK water regulator Ofwat publishes in its annual Service and delivery report the following table summarising the aggregate performance of companies in the industry under a range of key measures it has selected. Although primarily focusing on industry performance of companies, it provides Ofwat with a structured set of indicators and trend data which can be used to inform business planning decisions, for example, assessing whether deteriorating service performance requires Ofwat investigation or other actions. This example also promotes transparency by publishing summary information on key performance measures in one place and in a format clearly understandable to stakeholders. Figure 3 Industry service performance data published by Ofwat (extract) Description % % % % % Properties at risk of low pressure Properties subject to unplanned supply interruptions of 12 hours or more Population subject to hosepipe bans Billing contacts not responded to (within five working days) Bills not based on meter readings Received telephone calls not answered within 30 seconds (no longer collected) Note: For presentational purposes an extract is shown. The full table also provides summary data for the two periods and Source: Ofwat, Service and delivery performance of the water companies in England and Wales , October 2009, Table 1 page 7 (extract). 5 Performance of Ofgem: A briefing for the House of Commons Energy and Climate Change Select Committee, NAO, February 2011 ( Last saved 23/11/11 15:00 7

8 Case example: attributing market outcomes to regulatory interventions 17. The UK telecommunications market structure is summarised in Figure 4. Figure 4 The Market and Regulatory Structure of Telecommunications Products and services are delivered by private companies direct to individual and business consumers, and the market is regulated by Ofcom. The telecoms market started as a monopoly, with British Telecom (BT) providing fixedline services. The present-day market has a number of providers of both fixed and mobile communications (including broadband), which are regulated by Ofcom. Ofcom no longer controls prices, but it does place conditions on BT to enable competitors to access its infrastructure. Ofcom uses informal approaches to regulation, such as voluntary codes of practice and publication of performance information, resorting to more formal regulation if necessary. It licenses companies (and others) to use the radio spectrum. The companies providing products and services are privately-owned. They compete with each other primarily on price, as well as the product offer. Potential advantages The model is cost-effective: it meets the objective of providing a service with reasonable minimum standards. Equity of service provision: services are widely available and accessible to most people. There are a reasonable number of providers, with no natural trend towards a monopoly and good safeguards against collusion. Potential disadvantages / risks It is a litigious environment, which can curtail Ofcom s ability to act quickly. There is potential for information asymmetries and therefore a risk that users can t make informed choices. 18. The statutory duties of the UK s telecommunications regulator, Ofcom, are primarily discharged through influencing the companies it regulates, and outcomes can be influenced by a wide range of external factors. In order to develop a holistic assessment of the regulator's performance, in our report 6 we sketched out a sphere of influence model (Figure 5) which takes account of the degree of control Ofcom has over a range of performance indicators. It works on the principle that a high-performing regulator would be achieving good results on all of the indicators over which it has most control, it would achieve good results on a majority of the indicators over which is has a reasonable degree of control, and good results on some of the indicators over which it has less influence. 6 Ofcom: The effectiveness of converged regulation, NAO, November 2010 ( Last saved 23/11/11 15:00 8

9 Figure 5: Sphere of Influence Source: NAO report, Ofcom: The effectiveness of converged regulation 19. Figure 5 is a visual illustration of this concept. The performance measures of interest are set out around the outside of the circle, and the marker relating to each shows the extent to which that indicator is within the regulator s control. In this case,. 1 = indicators more influenced by the regulator s actions than by other factors 2 = indicators about equally influenced by the regulator and by other factors 3 = indicators more influenced by other factors than the regulator s performance. 20. It is important to consider that a holistic assessment of performance clearly relies on judgements at several levels, for instance: - regarding the criteria for selection of indicators - assessment of degrees of influence, and - views on what counts as a good result on each indicator However, applied by a regulator, this type of approach has the advantage of being transparent to stakeholders about the need for making these judgements. It allows the regulator to engage with its stakeholders in making such judgements and setting thresholds for what will count as good performance when developing its work programme, and then to be held accountable against these standards. 21. Evaluative assessments that seek to attribute whether or not individual or collective regulations are working or achieving their intended outcomes, require close consideration of outcome measures and the extent of attribution causing any observed changes. This type of assessment relies heavily upon the indicators that determined the need for the regulation in the first instance. A clear evaluative framework is needed and robust data on costs and benefits necessary for the statistical analysis to attribute the outcomes achieved following implementation. Last saved 23/11/11 15:00 9

10 22. Post-implementation analysis informs policy makers and regulators in making future regulations and targeting these to the areas most likely to achieve success. Gary Coglianese, 7 Director, Penn Program on Regulation at the University of Pennsylvania Law School, has produced a framework that develops further the evaluation of regulation and regulatory policy, discussing possible measures for evaluating and monetising costs and benefits of regulation. Three types of evaluation, increasing in depth of analytical analysis, are discussed in his paper, considering the categories of measures for the evaluation of regulatory administration, behavioural compliance and outcome performance. 7 Cary Coglianese, Edward B. Shils Professor of Law And Professor f Political Science; Director, Penn Program on Regulation at the University of Pennsylvania Law School, Conference Paper Evaluating the Performance of Regulation and Regulatory Policy, September Last saved 23/11/11 15:00 10

11 Part 3 Assessing the value-for-money of a regulator The nature of value-for-money audit 23. The NAO defines good value for money as the optimal use of resources to achieve the intended outcomes. Value-for-money assessments are generally carried out ex post. Thus, to assess the extent to which an intervention, programme of activity, or organisation has delivered good value for money, we compare the actual outcomes achieved with one or more counterfactual sets of outcomes. The counterfactuals estimate whether results closer to those intended could reasonably been obtained by using the given resources differently, or whether the observed outcomes could reasonably have been achieved using fewer resources. (Note that the term optimal in the definition of good value for money does not mean the best across all possible worlds, but rather the best across all reasonable worlds.) 24. One of the challenges in assessing the value-for-money of a regulator is using information on outcomes in the market (which reflect both regulatory and wider factors) to provide information on the extent to which the regulator is achieving its intended outcomes. Moreover, apparently good outcomes (for current consumers), could coexist with poor regulation (not protecting the interests of future consumers) the situation in the financial sector before the credit crisis provides an example of this. The value-for-money audit of regulation needs to assess, therefore, both whether performance to date is good (a function of both market outcomes, and conduct of regulation) and whether risks for the future are being well managed (the conduct of regulation). 25. As the first stage in assessing the value for money of a regulator, the NAO looks at how the regulator measures its own performance. A performance measurement system should be - Focused on the organisation s aims and objectives; - Appropriate to, and useful for, the stakeholders who are likely to use it; - Balanced, giving a picture of what the organisation is doing, covering all significant areas of work; - Robust, in order to withstand organisational changes or individuals leaving; - Integrated into the organisation, being part of the business planning and management processes; and - Cost Effective, balancing the benefits of the information against the costs. 26. These principles are from guidance published by the NAO, the UK Treasury and others 8. The guidance states that a performance information system should have clear links between the high-level objectives of the organisation and its measures of success. For a regulator, those high-level objectives will be set in statute. 27. Good performance information is needed to underpin value-for-money conclusions on regulators. In our report on the UK communications regulator Ofcom, we found that there was evidence from some individual projects that Ofcom considers value for money in its decision making, rather than purely seeking to minimise costs. However, in the absence of a clear articulation of its intended outcomes, or the linkages between its inputs and outputs and those outcomes, there was still a gap between the way Ofcom assessed its own performance and our assessment of value for money. It was, therefore, not possible for us to conclude on the extent to which Ofcom has delivered value for money. We concluded that this would remain the case until Ofcom developed a consistent capacity to link costs and activities to outcomes in a way that can help stakeholders assess its performance. 8 Choosing the right FABRIC: a framework for performance information, HM Treasury, Cabinet Office, National Audit Office, Audit Commission and Office for National Statistics, 2001 Last saved 23/11/11 15:00 11

12 28. The UK Parliament, through the Public Accounts Committee, agreed with this finding, saying that Ofcom should indicate in advance what success would look like 9. Ofcom s latest Annual Plan was much clearer about its intended outcomes and Ofcom has made a good deal of progress in reporting the linkage between its outputs and outcomes. The latest Ofcom Annual Report 10 has clear links to the objectives set out in the Ofcom Annual Plan, their performance against these priorities and provides an easy to understand level of detail. Ofcom has set out what they said they would do, what actually happened to achieve this and the outcomes for consumers. 29. The UK Government recently commissioned a review 11 of the water regulator, Ofwat, which also concluded that good performance measurement was necessary to provide assurance to consumers that the regulator was delivering value for money. It reported that A number of respondents [to its call for evidence] felt that the arrangements to hold Ofwat to account for its decisions were inadequate and that a more effective process to oversee its activities was required. This is, in many respects, a wider question, applying to all economic regulators, and goes beyond the remit of the review. However, we feel that Ofwat could make it easier for Parliament to hold it to account by publishing a set of objectives, or desired outcomes, against which its actions could be assessed. This would also assist the National Audit Office in assessing Ofwat s value for money. 12 It recommended that Ofwat should develop outcome-based performance measures. One of these measures should focus on reducing the regulatory burden [on industry]. The NAO, in its independent role in scrutinising the value for money of the regulator, should be involved in monitoring Ofwat s performance against these measures In our report on the Office of the Rail Regulator, we concluded on Value for money that the regulator has significantly developed its analyses used in regulatory settlements and the methods it uses to judge Network Rail s efficiency. However, there are weaknesses in the Regulator s information on Network Rail s unit costs, and it needs to develop a better understanding of the reasons for the efficiency gap relative to more efficient operators. We judge that these limitations have not been critical to the pursuit of value for money to date, because of the scale of the efficiency potential revealed by the Regulator s external benchmarking. In order to make further improvements though, we believe that these weaknesses must be addressed promptly, to provide confidence that firstly, efficiency targets for the next price control period reflect Network Rail s true potential to secure value for money as the efficiency gap narrows; and secondly reported efficiency gains correctly reflect Network Rail s actual performance. 31. The Public Accounts Committee hearing in May 2011 discussed this report on the Office of Rail Regulation, and was followed by a more general hearing on rail in June. This also touched upon a wider independent review of rail, and the PAC 14 concluded that it did not believe that the Regulator exerted sufficient pressure on Network Rail to improve its efficiency, and that there is an absence of effective sanctions for underperformance in the system. The PAC recommended strengthening the role of the regulator, as well as for the NAO to have audit access to the rail network provider, Network Rail. 32. A common theme in the findings on VFM in recent NAO reports, largely on the regulators but also in regard to other government departments, is that even when good progress has been made to date, wider issues also need to be considered. Most notably, the changing 9 Ofcom: The effectiveness of converged regulation, Committee of Public Accounts: 20th Report , Feb The Office of Communications: Annual Report and Accounts for the period 1 April 2010 to 31 March 2011, Ofcom, July Department for Environment, Food and Rural Affairs, Review of Ofwat and consumer representation in the water sector, July Ibid., p Ibid., p Office of Rail Regulation: Regulating Network Rail's efficiency, Committee of Public Accounts: 41st Report , July Last saved 23/11/11 15:00 12

13 delivery mechanisms used by policy makers, such as the wider use of market based systems and also these models present policy makers with challenges monitoring and evaluating progress and success. These evolving systems need to be robust enough to address both current and future needs. This trade-off is evident in the NAO report on user choice and social care (discussed in Part 2), the NAO concluded on VFM that evaluations suggest that, most social care users have gained improved well-being and control from using personal budgets, and some local authorities report they have led to efficiency improvements. The UK Government intends to extend personal budgets to all eligible users of social care by For this policy to deliver VFM in the future, sufficient oversight arrangements must be in place to ensure care markets operate effectively. At the moment, however, there are wide variations across local authorities in the amount of engagement, information and support offered to those people using personal budgets. No formal arrangements exist to monitor care markets nationally or regionally, these shortcomings must be addressed to ensure future VFM is secured. Conclusion 1. The NAO s work shows that, in the UK: - Parliamentary accountability for most of the economic regulators could be improved, with better-quality regulatory performance reporting. This includes translating the regulator s statutory duties into operational objectives, a clear line of sight between high-level objectives and lower-level measures, and a focus on outcomes. - The relationship between regulators and government departments is not always optimal, partially due to a lack of clarity over the exact role of the regulator, and the extent to which it should comment on policy matters. 2. Judgements on value for money in regulating infrastructure networks require: - a sound understanding of the regulatory environment, including the nature of regulated bodies and the tools including information and incentives available to the regulator; - knowledge of the outcomes of different regulatory approaches in similar environments; - a recognition that the distinctiveness of some regulatory environments prevents definitive value-for-money judgements, since no counterfactual evidence base is available; and, 3. Given the current austerity measures in the absence of such counterfactuals, an a priori framework for judging the characteristics which would be expected to be found in a valuefor-money regulator. Given the current austerity measures in the UK and the drive to cut costs in the public sector, Government departments are under pressure to achieve policy objectives with less funding than in the past using more dynamic delivery mechanisms that are largely untested in many sectors. Government policy may cut funding for services, for example transport, leading to greater costs for consumers who pay for these measures ultimately with higher rail fares. As scrutiny of user-funded services develops and increases, regulators will need to be able to demonstrate convincingly that they are delivering VFM. 4. Moreover, the policy/funding dilemmas facing the regulators have wider applicability across government. In the UK there is a significant shift away from command and control approaches to public service delivery. While it may not always be called regulation, there is a clear desire to change the nature of the relationship between the policy centre and delivery units, and move towards a more plural society with multiple centres of power. The centre will increasingly define a role based on setting broad standards; devolving decisionmaking responsibility; and influencing rather than directing behaviour. These are all characteristics of a regulatory relationship. Current discussions in the UK, particularly with Last saved 23/11/11 15:00 13

14 regard to the future shape of the health service, are looking to the lessons from the regulation of infrastructure industries, and in particular to what has been learnt about the introduction of competition in markets with the aim of lowering prices and bringing greater quality of service to consumers through the power of choice. Last saved 23/11/11 15:00 14