Performance measurement practice. The use of sanctions and rewards in the public sector

Transcription

1 Performance measurement practice The use of sanctions and rewards in the pubic sector

2 The foowing individuas contributed to this report: Sascha Kiess, Diane French, Nick Soan, Dan Vaance, Daniee Wiiams. The Nationa Audit Office scrutinises pubic spending on behaf of Pariament. The Comptroer and Auditor Genera, Tim Burr, is an Officer of the House of Commons. He is the head of the Nationa Audit Office which empoys some 850 staff. He and the Nationa Audit Office are totay independent of Government. He certifies the accounts of a Government departments and a wide range of other pubic sector bodies; and he has statutory authority to report to Pariament on the economy, efficiency and effectiveness with which departments and other bodies have used their resources. Our work eads to savings and other efficiency gains worth many miions of pounds; at east 9 for every 1 spent running the Office.

3 Contents Executive Summary 2 Part One Introduction 5 Part Two How sanctions and rewards are currenty used 12 Part Three The effectiveness of forma sanction and reward mechanisms 18 Annex 1 Methodoogy 38 Annex 2 Different types of sanctions or rewards 40 Annex 3 Guidance for deveoping and impementing sanction 45 and reward mechanisms Annex 4 Performance evers 50 Annex 5 Bibiography 54

4 Executive PART ONE Summary Executive Summary 1 This report is a review of the use of sanctions and rewards based on academic research and an NAO survey of their use in centra government. The NAO undertook this to inform the debate on incentivising pubic sector performance. Whie many commentators discuss pubic sector performance, they have not systematicay reviewed the use of sanctions for faiure or rewards for success. The purpose of this report is to bring together the evidence on the effectiveness of sanctions and rewards, summarise the resuts of our survey on their use in centra government, and to provide a practica guide on how to use them we. Key findings 2 Government bodies have to manage performance through deivery chains, often having imited contro over those in the chain. They usuay choose funding or reguatory instruments constructed so that they provide a degree of everage over performance. But such evers are often insufficient to secure a the desired resuts. That has ed to increased emphasis on sanctions and rewards, incuding the introduction of high profie schemes such as the earned autonomy of Foundation Trusts and the reward grant attached to achievement in Loca Area Agreements. At a ower eve, sanctions and rewards are being considered in reation to a variety of behaviour change initiatives, incuding sanctions for not recycing, financia incentives for businesses to move freight by water, and criminas being offered reduced prison sentences in exchange for peading guity as eary as possibe. 3 Despite this greater emphasis, there are few empirica assessments of the effectiveness of sanctions and rewards. The evidence that does exist shows that pubic sector agents organisations, teams and individuas do respond to incentives, and that those incentives can promote or hinder enhanced effort and performance, depending on how we they have been designed and impemented. 4 As we as reviewing evidence within the academic iterature, we conducted a survey of 145 government programmes to understand how sanctions and rewards are currenty used in the UK pubic sector. It identified three broad types of mechanisms: financia (e.g. bonuses or penaties); operationa (e.g. granting organisations greater or esser autonomy from inspection); and reputationa (e.g. the pubication of eague tabes). 2 The use of sanctions and rewards in the pubic sector

5 EXECUTIVE Summary Use of sanctions and rewards 5 Our survey showed: The overa use of sanction and reward mechanisms is ow. Ony around 40 per cent of those who responded to our survey reported using an expicit sanction or reward mechanism somewhere in their programme. To some extent, this refects a ack of famiiarity with the terminoogy: in a number of cases, foow-up to the survey uncovered that sanction and reward mechanisms were being used where none had been reported. Such responses, however, raise questions over the extent to which programme managers have systematicay considered the use of incentives. The use of sanction and reward mechanisms is more common in service deivery programmes. Neary 60 per cent of UK government programmes that deiver services reported using expicit sanction or reward mechanisms, compared to just 25 per cent for programmes that aim to buid capacity, and 21 per cent that seek to infuence outcomes. More sanction and reward mechanisms are aimed at the organisationa eve rather than the team or individua empoyee eve. This refects, among other things, the intrinsic difficuties of measuring the contribution of individua empoyees to many pubic sector outcomes. Financia and reputationa mechanisms are far more common than operationa ones. Our survey identified 26 financia mechanisms, 21 reputationa, but ony seven operationa. Effectiveness of sanctions and rewards 6 There have been few pubished empirica studies of the effects of sanctions and rewards in the pubic sector, and those that have been done focus on US institutions. There have been a number of further reviews and studies which provide some reevant evidence, however, as we as practitioner or expert judgements. Our survey showed a simiar picture. More than 60 per cent of those using sanctions or rewards (22 out of 35) beieved that their use had improved performance. Interviews with the respondents showed, however, that these opinions were based on subjective judgements. We found no quantified evidence of the effect of sanctions and rewards on eves of performance for the programmes in the survey. 7 If the evidence on the effectiveness of sanctions and rewards is imited, it does permit a number of observations: Peope in the pubic sector respond to sanctions and rewards, if they regard them as significant. Athough pubic sector workers may we have a variety of motivations to perform we, financia rewards can improve performance. If stakehoders have a mix of sef-interested and atruistic motives, then the design of sanction and reward mechanisms must recognise that mix, to avoid reducing overa motivation by focusing on one aspect to the excusion of the other. Because performance in the pubic sector has a number of dimensions, it is important that the definitions used for sanctions or rewards capture a significant aspects of performance, to minimise the scope for perverse effects. If performance is difficut to measure, or where the contribution of a stakehoder is difficut to attribute to performance, then sanction and reward mechanisms which have dramatic financia or persona effects shoud be avoided. Good practice checkist 8 The tabe beow summarises the main practices which emerge from the research and guidance. More comprehensive guidance is avaiabe at Annex 3. The use of sanctions and rewards in the pubic sector 3

6 Executive Summary Tabe 1 Guide for incentivising a deivery chain Objective 1 Ceary define a measurabe objective. Map the strategy 2 Identify the activities and the outputs that are needed to achieve the objective (see the exampe Strategy Map in figure 1). [p. 7 and 47]. Map the deivery chain 3 Identify and engage with the key stakehoders, map the performance evers and incentives in use (see the exampe Deivery Map in figure 2). [p. 8 and 48]. 4 Use the deivery map to undertake a stakehoder anaysis. Identify where the main barriers are, and the evidence on the effectiveness of the current programme, the performance evers, and their associated sanction or reward mechanisms (see Stakehoder Anaysis tempate in Annex 3). [p. 49]. Designing sanction and reward mechanisms 5 Understand the fu range of sanction or reward mechanisms avaiabe financia, reputationa and organisationa (see Annex 2). [p. 40]. 6 Assess the motivations of the key payers. Assess the baance between financia and atruistic motivations, and whether they operate at organisationa, team or individua eve. [p. 26 and 40]. 7 Identify outcomes over which the payers have contro or significant infuence. [p. 27]. 8 Link the incentives to performance measures which ead to the desired (ong-term) outcomes in a predictabe way. A singe measure may not capture the reevant aspects of performance, but any set of measures must be kept manageabe. [p. 28]. 9 Ensure the rewards and sanctions are cost-effective. Where they invove financia eements, it may be sensibe to mode the operation of the system to hep define appropriate vaues or ranges. [p. 30]. 10 Introduce safeguards to prevent unintended behaviours. Approaches can incude maintaining a degree of independence in performance assessment and vaidating key performance measures. [p. 32]. Measure performance 11 Deveop and impement data systems that coect and anayse timey and reiabe performance data. [p. 33]. Appy the sanction or reward 12 Deveop the expertise to appy the sanctions or rewards in a timey, consistent and transparent manner, especiay when using compex contractua arrangements with deivery agents. [p. 34]. Review effectiveness 13 Buid reguar interna and externa reviews of the effectiveness of the sanction or reward mechanism into the programme s overa performance management cyce. [p. 35]. Impement 14 Where possibe, phase in new sanction or reward mechanisms graduay, or on a piot basis, to identify and address any dysfunctiona behaviour. [p. 37]. 4 The use of sanctions and rewards in the pubic sector

7 Introduction PART ONE Introduction UK pubic sector context 1.1 This report contributes to the debate on incentivising pubic sector performance. It examines the various evers used by government to meet the ambitions set out in Pubic Service Agreements (PSAs) and other major programmes of work, and asks whether the effectiveness of these evers coud be enhanced by the use of expicit or forma sanctions and rewards. Whie there has been much commentary on pubic sector performance, there has been imited systematic review of the use of sanctions for faiure, or rewards for success, in achieving progress. 1.2 The Government s approach to pubic service reform since 1997 has changed the reationships between centra government, oca government, and the many other organisations and citizens invoved in deivering government objectives. Cabinet Office (2006) summarised it as comprising four eements: Top-down performance management, such as PSA targets, reguation/minimum standards, performance assessment, inspection, and intervention to tacke poor performance. Introduction of greater competition and contestabiity in the provision of services. Introduction of greater pressure from citizens (choice and voice). Measures to strengthen the capabiity and capacity of civi and pubic servants. 1.3 The Government has aso promoted the merits of ocaism the devoution of performance management to a oca eve, with ess centra dictation of priorities: Our ong term objective has aways been to match ambitious nationa standards with a vigorous oca autonomy and fexibiity whereby we maximise efficiency, equity and a personaised service for the patient, the pupi and the citizen generay. This opens up a chaenging agenda for the second stage of modernisation and reform. It incudes far more radica devoution of responsibiities from Whiteha as we give the roe of Whiteha a sharper focus. It aso demands greater transparency, proper audit and new incentives and an ambitious agenda to encourage oca autonomy and manageria fexibiity. (HM Treasury, 2004a). The use of sanctions and rewards in the pubic sector 5

8 PART ONE introduction 1.4 The Treasury goes on to argue that this devoution of performance management needs to be accompanied by stronger oca accountabiities and incentives for deivering improvements: Credibe incentives in the form of rewards and sanctions must be made avaiabe such that intervention by centra government ony takes pace in cases of cear under-performance that the intermediate tier has not been abe to correct. 1.5 Exampes of expicit sanction and reward mechanisms incude: contracts with private sector firms, where financia bonuses and penaties are specified for performance; tying the degree of externa inspection to performance, as in the case of the Audit Commission with Loca Authorities; and the use of pubic benchmarking, such as schoo eague tabes. In other cases, incentives are not reiant on expicit sanctions and rewards, such as the egisation that requires Loca Authorities which underperform against air quaity targets to pubish action pans. Deivery chains 1.6 Most government objectives cannot be deivered by government Departments aone, but require a compex network of organisations, incuding centra and oca government, government agencies, and bodies from the private and third sectors. In the iterature these inks have been anaysed by reference to Principa-Agent theory, which describes the issues faced by one party (the Principa ) when it needs another party (the Agent ) to deiver its objectives. If Departments are the Principa, then the Agents are the empoyees, organisations or citizens that Departments need to work with to deiver their objectives. NAO (2006) ists four basic types of inkages or reationships within these deivery chains. Different types of ink need to be managed in different ways. For exampe: Interna inks; where one part of the chain directy manages another. These can be managed through a standard management review process. The Principa sets objectives for the Agent, who then carries out the activities and reports progress back to the Principa. If performance is poor, the Principa works with the Agent to improve performance. However, this can be very resource-intensive. Contractua or reguatory inks; where one part of the chain defines through aw, and/or funding, how another part operates. These are often managed more at arms-ength, where the Principa formay deegates responsibiity for certain outcomes to the Agent. The Principa wi often use sanctions or rewards as a way of motivating the Agent to deiver on objectives, avoiding the need for resource-intensive performance management. Given the move towards ocaised performance management in the pubic sector, there has been an increase of interest in the potentia for using sanctions and rewards. Links of common purpose; where two bodies have parae missions to work towards the common good, such as between two government Departments, or between a Loca Authority and a charity. These rey more on partnership working and trust, as there is no cear Principa or Agent. However, these inks can be strengthened through more forma instruments, such as Memoranda of Understanding, in which parties set out their commitments to each other. As with contracts, they may incude Service Leve Agreements to carify and strengthen these commitments. Links to the wider community; where one part of the chain has no forma authority over the groups or individuas with which it wishes to work, and is therefore reiant on persuasion. Here there is a need to rey on softer evers, such as providing forma education or information as a means of persuading Agents to behave in a certain way. Advertising campaigns or pubic information eafets are commony used for this purpose. 6 The use of sanctions and rewards in the pubic sector

9 Introduction PART ONE Strategy and stakehoder mapping 1.7 It foows that designing effective incentives depends on a good understanding of the context, identifying the various performance evers avaiabe, understanding the motivations of the deivery agents, assessing the strengths and weaknesses of the evers, and identifying opportunities to strengthen the evers to achieve the desired eves of performance. These steps to panning the deivery of government programmes were formaised within Departments Deivery Agreements, which were pubished for each of the 30 Pubic Service Agreements under the government s 2007 Comprehensive Spending Review. The key eements in this process are summarised beow. 1.8 Figure 1 shows a generic strategy map for an objective. It shows how an organisation can take an objective, and identify the combination of activities and outputs that shoud hep to achieve it. It shoud be based on evidence, or at east on expert judgement, on what drives the outcome. This sort of mapping is designed to hep the owner of an objective create a programme structure that is ikey to achieve the objective in a cost-effective way. A strategy map can aso be used to structure the performance measurement framework, as it identifies the outputs and outcomes that are important to measure. These indicators are aso ikey to feature in any sanction and reward mechanisms. 1.9 Having identified the activities needed to deiver an objective, suitabe organisations and stakehoders can be chosen to perform those activities. It is then hepfu to map the reationships between them incuding the performance evers that are used and any sanctions or rewards that are attached to them. This type of deivery map can hep identify where there is insufficient everage over performance, and where introducing expicit sanctions or rewards is ikey to have the greatest impact. It can aso be used to ensure coherence of incentivisation across the deivery chain, and as a basis for considering the ikey motivations of individua organisations or stakehoders. 1 Strategy Map Objective Intermediate outcome 1 Intermediate outcome 2 Intermediate outcome 3 Intermediate outcome 4 Output 1 Output 2 Output 3 Output 4 Output 5 Output 6 Activity 1 (Organisation 1) Activity 2 (Organisation 2) Activity 3 (Organisation 3) Strong infuence Moderate infuence Weak infuence Source: Nationa Audit Office The use of sanctions and rewards in the pubic sector 7

10 PART ONE introduction 2 Deivery Map Funding Performance Management Organisation 1 (Activity 1) Funding Pubished performance eague tabes Department Organisation 2 (Activity 2) Education Contract Organisation 3 (Activity 3) Education Target popuation Financia bonus Taxation Externa infuences Performance ever Sanction or reward Strong infuence Moderate infuence Weak infuence Source: Nationa Audit Office 1.10 Tabe 2 iustrates some of the evers which exist and how they might be used with, or without an expicit sanction or reward. (Further discussion of performance evers is avaiabe at Annex 4). The main motivations for using sanction or reward mechanisms incude: To improve the performance of individua units in comparison to overa performance for exampe, of an individua hospita reative to other hospitas. To improve the performance of a wider system for exampe, to improve the performance of the heath system to dea with the threat and occurrence of strokes. As a means of aocating resources to more efficient providers for exampe, to enabe funding to be aocated to a number of different providers based on their performance. (Propper and Wison, 2003). 8 The use of sanctions and rewards in the pubic sector

11 Introduction PART ONE Tabe 2 Performance evers Performance ever Without expicit sanctions or rewards With expicit sanctions or rewards Taxation which, as we as raising government revenue, can be an effective means of incentivising certain behaviours by increasing or decreasing the costs associated with them. Taxes on acoho, tobacco and fue hep to raise income for the government. Because the amount of tax individuas pay for these goods is directy inked to the amount they consume, it has an intrinsic effect of motivating consumers to buy ess, and so can be used as a performance ever. Lower rates of Vehice Excise Duty are offered to Heavy Goods Vehice operators who achieve target eves of emissions and obtain a Reduced Poution Certificate. Pubishing performance data which, in a competitive environment, heps service users make an informed choice, and so can be used to encourage competition between individuas and organisations to perform better. Legisation which sets the overa ega framework for deivery chains and can enabe the use of other evers. Reguation and standards which can be used to specify the quantity, quaity and/or type of service that providers shoud offer users. Inspection and review which is used to check the quaity of deivery or the standards of performance. Targets which, combined with the pubication of associated performance indicators, are intended to focus attention on key outcomes and to enhance transparency and accountabiity. Funding which can be inked to various measures, such as past performance, organisationa capacity, and the achievement of targets. Train Operating Companies are required to pubish their environmenta poicies, as we as their performance against key envinromenta indicators. By aowing greater pubic scrutiny, it is hoped to encourage the companies to deveop sound poicies, and perform we against them. The 1995 Environment Act requires Loca Authorities to assess their air quaity. There are no forma sanctions if poution is excessive, but the area wi be designated an Air Quaity Management Area and an action pan must be drawn up. Ofgem sets price caps for eectricity and gas companies, which incentivises efficiency and contributes to the government s objective of maintaining the competitiveness of the UK s energy markets. Pubication of the resuts of Department Capabiity Reviews provides greater transparency of the performance of individua government Departments. Loca Authorities were given efficiency targets foowing the Gershon Review. Performance against these targets was monitored but there was no forma consequence for non-achievement. Centra government Departments are aocated funding dependent upon their performance in the two previous years, but the ink between performance and funding eves is not made expicit. Converting basic performance information into composite indicators such as Loca Authority assessments or hospita star ratings makes it cear who is performing better or worse overa, and adds further motivation to perform we. Loca transport authorities are entited to estabish statutory Quaity Partnership Schemes with bus operators to require them to provide services of minimum standard to ensure contract renewa. The Environment Agency can withdraw the operating icences of arge-scae industria pouters such as car producers and hauiers if they fai to compy with minimum standards. Linking Loca Authorities Comprehensive Performance Assessment scores to the degree of intervention and inspection by the Audit Commission. Pubishing the ranking of schoos against the target for getting 60 per cent of pupis aged 16 to achieve the equivaent of five GCSEs at grades A* to C. Ranking schoos against target achievement rewards higher performers by attracting more pupis, bringing greater revenue. Reward grants for Loca Authorities, such as the Panning Deivery Grant the size of which depends on performance against expicit targets. The use of sanctions and rewards in the pubic sector 9

12 PART ONE introduction Tabe 2 continued Performance evers continued Performance ever Without expicit sanctions or rewards With expicit sanctions or rewards Interna performance and staff management organisations throughout the deivery chain use a variety of interna performance mechanisms (such as risk management) and staff management systems (such as annua performance reviews) to improve capacity and performance. Many staff appraisa systems use assessments of individuas performance against non-quantified objectives to manage and motivate the desired behaviour. Performance Reated Pay for Permanent Secretaries is partiay inked to performance against specificay quantified diversity targets. Contracts which formay deegate responsibiities for achieving certain eves of outputs or outcomes to third parties. Partnerships and Memoranda of Understanding which can be estabished where two or more bodies in a deivery chain have inks of common purpose or parae missions. Education and information which can be used to reinforce the effectiveness of other evers, and to change attitudes and behaviours. Informa incentives can be created by the threat of faiing to extend a contract beyond its minimum period, such as Capita s contract with the Department for Chidren Schoos and Famiies to run the Nationa Strategies programme. Amost a Pubic Service Agreements are now shared between Departments, and formaised through jointy agreed deivery pans. Pubic information campaigns such as the drink-driving campaign are aimed at incentivising a change in behaviour. Transport for London contracts contain financia bonuses and penaty deductions for bus operators, reated to their performance against agreed metrics. These types of evers typicay do not incude forma sanctions and rewards, uness both parties agree to introduce a contractua eement. Not appicabe. Educating and informing organisations and individuas is ikey to support one or more of the other performance evers described above. Study objectives and scope 1.11 Against this background, which impies that a degree of deivery chain anaysis wi have been performed, the objectives of this study are: to identify and cassify the sanction and reward mechanisms used in both the pubic and private sectors; to assess the evidence on the effectiveness of different sanction and reward mechanisms in different contexts; and to provide some guiding principes to be considered when designing and impementing a system of sanctions or rewards We use the term incentives to incude a mechanisms by which organisations and individuas are motivated to perform. We differentiate between two forms of incentives: performance evers such as the existence of targets or the routine pubication of performance data, which may impicity motivate performance but are not deiberatey or primariy designed for sanctioning or rewarding purposes; and sanctions and rewards which are designed to strengthen these performance evers by penaising or rewarding certain eves of performance or discrete actions. We identify three types of sanctions and rewards: 10 The use of sanctions and rewards in the pubic sector

13 Introduction PART ONE financia for exampe bonuses or penaties based on performance; operationa this incudes granting organisations greater or esser autonomy from inspection, or promoting individuas on the basis of their performance; and reputationa such as the pubication of eague tabes or offender photographs. This study focuses on the use of sanctions and rewards, but many of the principes and techniques are aso reevant to the performance evers that underpin them. Our approach 1.13 Our approach has combined: desk-based research on the evers used in the pubic sector, focusing particuary on the use and effectiveness of sanction and reward mechanisms; a survey of centra government Departments on the sanction and reward mechanisms used in some of the government s key programmes; and structured interviews with poicy eads in Departments, and key deivery officias, for eight of the government programmes that responded to our survey. Annex 4 describes in detai our methods and the process of seecting the case studies. The use of sanctions and rewards in the pubic sector 11

14 PART two ONE How sanctions and rewards are currenty used How sanctions and rewards are currenty used 1. Government programmes were identified through the Pubic Service Agreements in operation in winter Whist the PSA framework changed in Apri 2008, the major programmes of Departments have continued; often now defined as Departmenta Strategic Objectives. Overview 2.1 This Part describes how some of the government s key programmes are deivered, and presents the resuts of our survey of government programmes. We use these resuts to highight, describe and interpret the various uses of sanctions and rewards. The use and non-use of sanctions and rewards 2.2 To understand how sanctions and rewards are currenty used, we surveyed 145 government programmes 1. We received 91 responses a response rate of 63 per cent. Of these, 56 (61 per cent of those that responded) reported that they do not use any forma sanction or reward mechanisms. We categorised the surveyed programmes according to whether they were primariy deivered through services, by infuence (e.g. to change peope s behaviour), or by buiding capacity. Figure 3 shows that expicit sanctions or rewards are more ikey to exist within deivery chains that are focussed on deivering services. 3 How frequenty sanctions and rewards are used for different deivery methods Number of government programmes Infuencing Not used Used Deivering services Buiding capacity Source: Nationa Audit Office survey of Departments, and anaysis of UK government deivery chains 12 The use of sanctions and rewards in the pubic sector

15 How sanctions and rewards are currenty used PART two 2.3 This pattern is probaby due to the difficuty in attributing actions to outcomes. It is easier to observe, measure, and therefore incentivise the contribution that individuas and organisations make when deivering a service, than when they are infuencing outcomes. Tabe 3 ists a sampe of sanction and reward mechanisms that are used in UK pubic sector programmes. Tabe 3 Exampes of sanctions and rewards used in the UK pubic sector Renewa of Air Quaity Grants to Loca Authorities is dependent upon achievement of the targets specified in the origina grant. Deductions from out-of-work benefit payments by the Department for Work and Pensions in the event of non-compiance with mandatory activities, such as attending job interviews. Bonus funding above a standard amount through the Panning Deivery Grant to Loca Authorities, based on performance criteria reated to the speed of processing appications. Financia incentives offered by the Department for Work and Pensions to recipients of incapacity benefits or one parent benefits, to hep them move into and remain in work. Bonus of up to 7.5 per cent paid to teams in Jobcentre Pus on achievement against five performance criteria. Green Fag Award for good management of parks and open spaces, given by the Department of Communities and Loca Government to Park Management and Maintenance. Decreased oversight and greater freedom from centra management given to hospitas who attain Foundation Trust status by meeting certain performance standards. (For a wider ist of exampes see Annex 2). 2.4 Survey responses showed that more than haf of government programmes did not use expicit sanctions or rewards. However, it is possibe that this is an overstatement, as further research into those programmes uncovered a number of ow profie mechanisms which had been overooked by those competing the surveys. We seected eight programmes that responded to our survey for further research four that reported using sanctions or rewards, and four that did not (Annex 1). Of the four programmes that reported not using sanctions or rewards: The first exampe occurred as part of the Department for Communities and Loca Government s objective to reduce homeessness. It reported funding, partnership working, and good reationships with Loca Authorities as primary evers, and fet that the strength of these primary evers mitigated the need for forma sanctions and rewards. However, one Loca Authority cited a eague tabe which is circuated by the Department, and ranks a Authorities according to the number and percentage of famiies they have managed to move from temporary to permanent accommodation as a big incentive: Athough the performance measures aren t perfect, nobody wants to come ast. In the second exampe, the Department for Environment Food and Rura Affairs is renegotiating Service Leve Agreements with its three partner agencies Natura Engand, the Regiona Deveopment Agencies, and the Rura Payments Agency. These Service Leve Agreements wi enabe the Department to recaim greater eves of funding if the required eves of performance are not met. The use of sanctions and rewards in the pubic sector 13

16 PART two How sanctions and rewards are currenty used 2. Reserve powers aow Secretaries of State to intervene in Loca Authorities. The third programme we examined cited the existence of a strong primary ever in the form of egisation as a reason for not yet having introduced expicit sanctions or rewards. The Department for Environment Food and Rura Affairs has a egisative ever to improve air quaity through the Environment Act. If Loca Authorities exceed certain concentration eves in any of the seven poutants, they need to produce a more detaied assessment of the probem. That may ead to a Loca Authority having to produce an Air Quaity Action Pan, with annua progress reports. Athough the Department has no forma enforcement powers as a reguator, it has the potentia to use reserve powers 2 in the case of non-compiance or where it considers that further measures are required. In practice these powers are amost never used, being seen as an extreme step in the context of devoved government. The Department notes, however, that the quaity and speed with which the Air Quaity Action Pans are produced, and soutions impemented, are variabe. The egisative ever aone, athough strong in principe, has not in practice ed to consistenty good performance. The Department is considering the scope to use further sanctions or rewards. The fourth programme reates to the Home Office objective to improve the perceptions of the pubic about drug misuse in their area. Perception measures can be used in two ways: as proxies for a more substantive outcome; or as outcomes in their own right. Initiay, the Department viewed pubic perception of the drug probem as a proxy for the probem itsef. It chose not to use any sanctions or rewards around the proxy measure, primariy because at the time perceptions were not a high priority, but aso because of the risk of creating incentives to improve perceptions in ways which did not reduce the underying probem. Subsequenty, the Department came to view perception as an outcome in itsef. To achieve a sustained improvement in perceptions, research shows that the substantive issues sti need to be addressed whist aso engaging oca communities in deveoping a shared understanding of the probems, deveoping oca soutions and communicating the outcomes of oca action. It has now incorporated the perception measure in the arge set of indicators used to provide sanctions and rewards for Loca Authority performance. This approach recognises the intrinsic vaue of improved perceptions, whie reducing the ikeihood of perverse effects by its incusion with a broader set of measures. 2.5 One of the issues surrounding the extent to which sanctions and rewards were used concerned the baance between centra direction and oca choice. In moving to more devoved decisionmaking, the Treasury identified an increased opportunity to incentivise performance, incuding the use of sanctions and rewards. This position is supported by the iterature and the ogic of incentivising, rather than directy managing, performance. In some cases, however, Departments reported that this devoution of priorities and budgets reduced their abiity to use sanctions and rewards to incentivise the deivery of nationa priorities. This highights the impicit tension in setting nationa priorities that can be impemented at oca discretion. To manage this tension centra government needs to deveop information systems to anayse whether oca needs have been accuratey determined and use this information to infuence oca bodies. 2.6 The cases where sanctions and rewards had been used are discussed throughout the remainder of this paper. The foowing exampe shows how sanctions and rewards have been added to primary evers to address a specific performance concern one of the more common ways they have been brought into use. The Nationa Treatment Agency is a key stakehoder in the government s objective to reduce the harm caused by drugs. For many years, it and the Department of Heath used the provision of guidance, the distribution of ring-fenced funding, and performance management as its primary performance evers. However, a review of unit costs showed a substantia variation in the average 14 The use of sanctions and rewards in the pubic sector

17 How sanctions and rewards are currenty used PART two amount of money being spent on each patient treated a variation that coud not be expained by oca circumstances, and therefore demonstrating a variation in performance. The Agency and the Department of Heath concuded that the evers used thus far coud be compemented by additiona incentives through the aocation process to generate a change in efficiency. For the panning round, the Department of Heath changed its funding formua towards a more eve contribution per individua treated. This has the effect of rewarding the more efficient areas, and sanctioning the ess efficient. This exampe demonstrates how two organisations have gone through a process of identifying a desired outcome, understood what motivates the deivery chain, measured the performance of the deivery chain, and used an incentive mechanism to address a shortfa in this case, the variabe efficiency of the deivery bodies. 2.7 The discussions we had with stakehoders as part of this exercise yied a number of propositions about use and non-use of sanctions and rewards: There may be occasions where the strength of the primary evers over performance, or the nature of the performance measures avaiabe, mean that sanctions and rewards are not needed, or not desirabe. But these issues have rarey been anaysed in depth. In the course of running a programme of work, Departments frequenty find the evers avaiabe to them insufficient to obtain the desired eves of performance. Sanctions and rewards do not aways need to be high-intensity financia bonuses or penaties, or widey pubicised programmes of naming and shaming. Even subte mechanisms, such as the reputationa impact of internay pubished benchmarking data, can have an impact. Which performance evers sanctions and rewards support 2.8 Our survey shows that targets are the singe most common performance ever in deivery chains that use sanctions and rewards (Figure 4), with a fairy equa use of contracts, partnerships, reguation and egisation. Types of sanctions and rewards used 2.9 From our review of the iterature and our own survey work we have identified three types of sanctions and rewards: Financia these are perhaps the most expicit, in which bonuses are offered, or fines imposed, depending on the Agent s performance. Reputationa these try to motivate behaviour through exposing Agents performance. This can be done through, among other things, the pubication of eague tabes, the giving out of awards, or naming and shaming. Operationa these can incude the amount of audit or inspection burden, or the amount of operationa freedoms granted to an Agent. 4 Legisation 13% Programmes that used sanctions or rewards aso used the foowing evers Other 7% Education 6% Targets 29% Whie in principe sanctions and rewards can hep resove tensions between centra and oca priorities, in practice the abiity to use them depends on the precise nature of wider funding and measurement arrangements. Reguation 15% Partnerships 15% Source: Nationa Audit Office survey of Departments Contracts 15% The use of sanctions and rewards in the pubic sector 15

18 PART two How sanctions and rewards are currenty used 3. Tough action against underperforming hospitas and trusts. Press Reease. Department of Heath. 4 June Financia and reputationa sanctions or rewards are the most common, and very itte use is made of operationa mechanisms (Figure 5) athough the Department of Heath has recenty announced the intention to make it easier for the NHS Chief Executive to impose operationa sanctions on underperforming Trusts. These woud aow the NHS Chief Executive to remove Trust management teams and chairs who fai to meet certain minimum safety, quaity and financia management standards, without having to make arge payoffs Within the broad categories of financia, reputationa, and operationa, we have found many different types of sanction and reward mechanisms. Tabe 4 ists some of the different types of mechanisms we have encountered specific exampes of these different mechanisms in practice are avaiabe in Annex Reputationa mechanisms are more ikey to be used when infuencing the deivery chain or buiding capacity, but financia mechanisms are by far the most common form for service deivery, making up 76 per cent of the mechanisms used in that category (Figure 6). Who uses sanctions and rewards 2.13 We identified four genera categories of stakehoders in a standard pubic sector deivery chain: centra government Departments; intermediary bodies such as Strategic Heath Authorities and reguators; deivery bodies such as hospitas and schoos; and citizens or participants often grouped by demographic, such as schoo pupis or benefit caimants. 5 Tabe 4 Category Financia The types of sanction and reward mechanisms used Number of sanction and reward mechanisms Reputationa Operationa Financia Reputationa Source: Nationa Audit Office survey of Departments Categories of sanctions and rewards Type of sanction or reward mechanism Contract or grant renewas. Hoding back payments. Absoute bonuses or penaties based on reaching certain performance threshods. Unit-based bonuses or penaties that vary continuousy according to the eve of performance. Discretionary ad hoc bonuses, in the form of cash or vouchers. Sharing profits or cost savings. Reduced charges, such as tax concessions. Pubic scoring and ranking, such as schoo eague tabes. Pubic recognition, through awards or kitemarks. Varying the eves of oversight through audit and inspection. Operationa Direct interventions, such as interimmanagement, dismissa or promotion. 16 The use of sanctions and rewards in the pubic sector

19 How sanctions and rewards are currenty used PART two 2.14 Our survey (Figure 7) shows that most of the sanction or reward mechanisms are used by intermediary bodies to infuence deivery bodies. This pattern is understandabe, given the reativey arge number of intermediary bodies reative to Departments of State Figure 7 does not show to what extent Departments themseves are incentivised to achieve their key objectives. The Treasury describes a number of evers that exist to motivate performance at Department eve: Financia. Performance against Pubic Service Agreements may have an impact on a Department s resource aocation in the next spending review, but the reationship between previous performance and future funding is not expicit or formaised. Staff performance. This makes up one eement of the appraisa process which infuences pay but in a subjective rather than an objective way. Performance management. Nominated Senior Responsibe Officers are made accountabe for each Pubic Service Agreement, and chair a Deivery Board within each Department. They then report to their Cabinet Committees, as we as the Prime Minister s Deivery Unit. Reputationa motivations. Performance is reported in Department Annua and Autumn Performance Reports, which are scrutinised by Ministers and Department Seect Committees, as we as by the pubic and media. Our survey shows that Departments, their intermediary bodies, and deivery bodies, use expicit sanctions or rewards aong the deivery chains, but we have found itte evidence of their existence at a Departmenta eve. 6 The types of sanction and reward mechanisms used for different deivery methods Number of sanction and reward mechanisms Source: Nationa Audit Office survey of Departments, and anaysis of UK government programmes 7 Infuencing Operationa Reputationa Financia Service deivery Buiding capacity The eve at which sanctions and rewards are appied Number of sanction and reward mechanisms Department on an intermediary body Department on a deivery body Intermediary body on a deivery body Source: Nationa Audit Office survey of Departments Deivery body on a citizen The use of sanctions and rewards in the pubic sector 17

20 PART three The effectiveness of forma sanction and reward mechanisms The effectiveness of forma sanction and reward mechanisms Overview 3.1 This Part summarises the research on the effectiveness of sanctions and rewards. Based on our iterature review, survey, and interviews with government programmes, we then provide guidance on how organisations can manage five key eements of a sanction and reward mechanism: design; measurement; appication; review; and impementation. What is meant by effectiveness 3.2 There are a number of dimensions in which improved performance may be measured: An increase in voumes: for exampe, the increase in the number of hip operations, or in the number of peope ooked after by Socia Services. An increase in efficiency: the same service may be provided at a ower cost, or more services provided for the same cost. An increase in user satisfaction: the quaity of the pubic service may increase, eading to higher eves of user satisfaction. 3.3 The first three of these dimensions aign with the standard three Es framework for assessing vaue for money: economy; efficiency; and effectiveness. A higher voume of inputs for the same cost represents an improvement in economy. Deivering the same, or more, service outputs for ower costs is an improvement in efficiency, whie an increase in user satisfaction is one way to measure improvements in the way outputs are transated into outcomes or effectiveness. 3.4 Equity objectives can sometimes confict with efficiency objectives or dimensions. For exampe, increasing equity in educationa outcomes may require skewing inputs heaviy towards the east abe pupis, whereas efficiency considerations woud suggest focusing teaching inputs on to pupis in the midde of the abiity spectrum. If equity concerns are important to the overa objectives of the pubic service which the sanction or reward mechanism is intended to support, then they must be considered at the time the mechanism is designed and reviewed. An improvement in equity: the distribution of users of a pubic service organisation is shifted towards the more disadvantaged for exampe, an educationa scheme may seek to target more users from minority groups. 18 The use of sanctions and rewards in the pubic sector

21 THe effectiveness of forma sanction and reward mechanisms PART three How effectivey sanctions and rewards are used 3.5 Concuding on whether specific sanctions or rewards have had direct impacts on performance is difficut. Our survey resuts showed that more than 60 per cent of those using sanctions or rewards beieved that their use had improved performance, whie the remaining 40 per cent were unsure. The various methods for reaching such a concusion in individua cases are discussed beow. There is a body of iterature on incentive mechanisms (see Annex 5) that is based on economic theory, and much of this is supported by a theoretica research base, often in the private sector. However, there is ess research evidence from the experience of the pubic sector. 3.6 There are a handfu of studies in the iterature based on detaied outcome data from randomised trias, amost a based on experiences in the United States. Nonetheess, the cear consensus based on the empirica evidence that exists is that pubic sector agents organisations, teams and individuas do respond to sanctions and rewards, and those incentives can promote enhanced effort and performance. Tabe 5 overeaf summarises exampes of the effective use of sanctions and rewards in the pubic sector. 3.7 In addition to the research summarised overeaf, two arge-scae productivity reviews ed by the Treasury in the UK, and the Government Accountabiity Office in the US, have found evidence to support the use of sanctions and rewards as an important aspect of improving performance. 3.8 The review of Devoving Decision-Making undertaken by the Treasury and the Prime Minister s Deivery Unit in 2003 investigated nationa and internationa best practice in heath, education, crime, and oca government through more than 20 site visits. They identified five critica factors that make performance measurement and management systems particuary effective, one of which was a transparent set of performance rewards and sanctions. 3.9 The Government Accountabiity Office (GAO) in the US has reviewed accountabiity for performance in the government s Grant Management Programme, which constituted neary 20 per cent of tota federa outays in Maximizing the extent to which grants achieve their ong-term performance goas and objectives is critica to successfuy addressing chaenges in the US. It concuded: As the cases we described iustrate, rewards and penaties are fundamenta toos to hep drive and motivate desired behaviors, but performance accountabiity mechanisms are not one size fits a; there is no universa transferabe mechanism appicabe to a programs. (GAO, 2006). The GAO found a number of opportunities to improve the design and impementation of performance accountabiity mechanisms Whie our survey showed that more than 60 per cent of those using sanctions or rewards (18 out of 22) beieved that their use had improved performance, further interviews highighted that these opinions were based on subjective judgements, and that there was itte quantifiabe evidence of the direct ink between the sanctions and rewards and the eves of performance. For exampe, the Department for Communities and Loca Government points to the fact that now more than 80 per cent of Loca Authorities are achieving their targets on the timeiness in processing panning appications, from ony 20 per cent in However, the Department does not know the reative effects each of a number of sanction or reward mechanisms had on this performance, or, other than anecdota evidence, whether this has had any detrimenta effect on the overa quaity of the process. The use of sanctions and rewards in the pubic sector 19

22 PART three The effectiveness of forma sanction and reward mechanisms Tabe 5 A summary of exampes of the effective use of sanctions and rewards in the pubic sector Type Financia (Organisations) The Lewin Group et a, 2003 Description of mechanism and research Mechanism: An incentive system in the US estabished by the Chid Support Performance and Incentive Act of 1998 provides incentive payments to states that meet one or more of the Act s five outcome-based performance goas and associated targets, and penaises states that fa beow threshod eves in certain areas. Research: A statistica regression anaysis to expain changes in state incentive payments during the transition to the new system in nine American states. Financia (Teams) Burgess et a, 2005 Mechanism: A UK scheme known as the Makinson Incentive Scheme in which a bonus payment was given to teams responsibe for VAT coection within the former HM Customs & Excise. Research: A controed experiment which compared the performance of two teams that quaified for bonus payments, with a team that did not quaify. It used statistica techniques to remove the effects of uncontroabe factors, such as staffing and oca market conditions. Financia (Teams) Burgess et a, 2004 Mechanism: Team-based financia incentives in the UK s Jobcentre Pus, covering five different targets incuding job pacements and customer service. Research: Used data from the Agency s performance management system and personne records, pus matched abour market data over the course of the one year piot scheme. Financia (Teams) Burgess & Ratto, 2003 Mechanism: Financia payments to a members of staff incuding teachers, ceaning staff and secretaries in highy performing US schoos. Payments were based on a statisticay-derived measure of test score gains which removed the impact of externa factors, such as pupis socioeconomic backgrounds. Research: Anaysis of data for schoos in six arge cities in Texas between 1991 and 1995, comparing those that used bonus payments to those that did not. Financia (Teams and individuas) Kahn et a, 2001 Mechanism: Performance-reated pay in the Braziian tax coection authority. This invoved the payment of financia incentives based on individua and team performance in detecting and fining tax evaders. Research: Used six years of data in ten tax regions three years before the reform and three years after. Used statistica modeing to separate out the effect of the bonus scheme from other macroeconomic and poicy changes. 20 The use of sanctions and rewards in the pubic sector

23 THe effectiveness of forma sanction and reward mechanisms PART three Assessment of success A review of the new incentive system in a sampe of nine states found that the median score on each of the five performance measures increased during the time the incentive system was impemented. Reasons for success Performance measures: A survey of states showed that the majority approved of the performance measures that were being used to sanction and reward them, as we as the threshod eves of performance. Impementation: States aso fet that impementing the scheme over a period of three years heped them to make a the necessary management changes. Both the teams that quaified for the bonus payments spent more time, and recovered more yied, on a trader groups than the team that did not quaify for the bonus. There is no discussion about why the scheme was successfu. The resuts suggest that the scheme was effective in raising job pacements in certain contexts, though the overa average effect was cose to zero. The impact of the incentive scheme was greatest in sma offices and in districts with fewer offices, but had a negative effect in arger offices. Team size: Setting incentives at a sma rather than arge team eve heps to overcome the free-rider probem, as it is easier for peers to monitor each other s performance. It aso found that the quantity of output increased, but that the scheme had itte effect on the quaity of service. It suggests that the use of incentive pay is much more cost-effective than a genera pay rise or increasing the number of staff. The resuts were generay positive, in that pass rates appeared to increase faster in a city that used financia incentives than in those cities that did not. There is no discussion about why the scheme was successfu. The findings show that the scheme had a dramatic effect. Fine coections per inspection were 75 per cent higher than in the estimated counterfactua, with impact ranging from 19 per cent to 145 per cent between regions. Leve of reward: The amounts invoved were substantia, frequenty providing bonuses over twice the average annua saary. The use of sanctions and rewards in the pubic sector 21

24 PART three The effectiveness of forma sanction and reward mechanisms Tabe 5 continued A summary of exampes of the effective use of sanctions and rewards in the pubic sector continued Type Description of mechanism and research Financia Mechanism: (Individuas) Atkinson et a, 2004 Performance-reated pay for teachers in Engand based on pupi scores. Research: A comparison of teachers who were eigibe for bonuses against those that were not. It used statistica regression techniques to take account of uncontroabe factors. Financia (Individuas) Dearden et a, 2004 Mechanism: The Education Maintenance Aowance invoved Engish Loca Education Authorities paying a means-tested benefit to year ods who remained in fu-time education after year 11 when education ceases to be compusory. Research: Anaysed the participation rates of year ods in neary 7,000 neighbourhoods during a one year piot of the scheme, comparing participation rates of those pupis who were eigibe for the bonus payments to those that were not. Reputationa Hibbard et a, 2003 Mechanism: Pubic reporting of performance. A report of hospita performance in the US caed QuaityCounts is pubished and circuated to the pubic. Research: A controed experiment, in which one group had the report circuated to the pubic, one group had the report just made avaiabe to themseves, and the fina group received no report at a. Operationa Propper et a, 2007 Mechanism: Making performance against waiting ist targets a key eement of the Engish NHS performance management system, with faiing managers being sacked, and successfu Trusts given greater autonomy. This was accompanied by additiona performance evers in Engand, such as escaating targets, and the pubication of performance. Research: This was a natura experiment based on the fact that Scotand and Engand have simiar heath systems, and so the key difference in performance is assumed to be due to the differences in how they are managed in this case, the fact that there were stronger sanctions and rewards in Engand than in Scotand. 22 The use of sanctions and rewards in the pubic sector

25 THe effectiveness of forma sanction and reward mechanisms PART three Assessment of success On average, the scheme increased pupi scores by haf a grade. However, it is not known whether this improvement was at the expense of other aspects of teaching. Reasons for success There is no discussion about why the scheme was successfu. The impact of the subsidy is quite substantia, with initia participation rates (at age 16) being around 4.5 percentage points higher. Fu-time participation rates one year ater are found to have increased by around 6.4 percentage points. Understanding the Agents: The presumption of the poicy makers has been that ow eves of participation are due to financia constraints, and so financia rewards are required to improve them. This is supported by the finding that the effect of the scheme is argest for chidren coming from a poorer socioeconomic background. Quaity improvement efforts among the pubic-reporting hospitas were significanty greater than in hospitas given ony private reports. This provides strong evidence that making performance information pubic stimuates quaity improvement in the areas where performance is reported to be ow. Understanding the Agents: The report is viewed as affecting the hospita s pubic image, and therefore is a key motivator to improve areas which need improving. The natura experiment aowed an amost direct evauation of the incentives. However, the study is unabe to determine the reative impacts of each sanction or reward. Leve of sanction or reward: NHS managers were sanctioned more harshy (by being sacked) but aso rewarded more (through greater autonomy) in Engand than in Scotand. Accompanying performance evers: Waiting ist targets were made tougher, and performance made pubic in Engand, but not in Scotand. The use of sanctions and rewards in the pubic sector 23

26 PART three The effectiveness of forma sanction and reward mechanisms 4. Athough 35 deivery chains reported that they use expicit sanctions or rewards, ony 18 answered a the questions concerning design, measurement, appication and review Our survey (Figure 8) aso showed that neary 40 per cent of deivery chains that used expicit sanctions or rewards do not know whether they have improved performance. 8 Have sanctions and rewards ed to improved performance? Not sure 39% No 0% Source: Nationa Audit Office survey of Departments Yes 61% 3.12 Whie there is imited evidence which aows us to isoate the direct impact that sanctions or rewards have on particuar dimensions of performance, a review of the iterature, combined with our own survey and interviews, identifies a number of essons on their effective management. We have divided these essons into the five key eements: designing the mechanism; measuring the incentivised performance; appying the sanctions or rewards; reviewing the resuts; and impementing or roing-out the system. Figure 9 overeaf iustrates this standard process for managing a system of sanctions and rewards. How to manage sanction and reward mechanisms effectivey 3.13 The foowing sections identify some of the practices within each of the five eements of a sanction or reward mechanism mentioned above, drawing out some genera concusions and guiding principes. Design 3.14 The design phase of any system is the most critica. Mechanisms need to be carefuy taiored to oca conditions to be effective. Our survey showed that neary a quarter of the 18 programme deivery chains who use expicit sanctions or rewards 4 were unsure about whether their mechanisms were designed using forma research or evidence. One of the key reasons given for this is that the systems were inherited, and the knowedge underying their design had passed from the corporate memory There is a we-estabished economics iterature which considers the optima design of incentives. The economic theory is framed in genera terms as a probem whereby a Principa needs an Agent to deiver its objectives, but the Principa cannot fuy contro (or even observe) the activity of the Agent. In the pubic sector, such reationships arise throughout deivery chains. This resuts in poicy eaders (the Principas) activey considering what evers they have to motivate the activities required by organisations, empoyees and citizens (the Agents) to deiver the desired outcomes In the simpest case, when the Agent s output is fuy observabe, and the vaue of the output is a perfect indicator of effort, the optima contract is for the Principa to pay the Agent the fu vaue of their output. But when the Agent s output aso depends on some random factor outside their contro, the Principa cannot infer the Agent s effort precisey. If the Agent is risk averse, then the optima incentive scheme consists of a fixed payment which acts as a risk premium and a margina payment proportiona to the output which acts as a reward. The optima baance between these eements known as the intensity of the incentive scheme depends on four factors: 24 The use of sanctions and rewards in the pubic sector

27 THe effectiveness of forma sanction and reward mechanisms PART three 9 Process for managing a system of sanctions and rewards 5. See Migrom and Roberts (1990) and Burgess and Ratto (2003). Design the system of sanctions and rewards Review the effectiveness of the system Measure performance against the defined parameters Appy the sanctions and rewards Impement the system Source: Nationa Audit Office the extent to which additiona performance resuts from extra effort, or the abiity of the Agent to infuence the outcome; Seect the right performance measure ensuring the incentivised outcome, or intermediate output, eads to the overa desired outcome. the accuracy with which activities can be measured; the Agent s toerance for risk; and Motivate a stakehoders. Take into account the team structures of the organisation. the Agent s responsiveness to incentives The key insights are that the more risk averse the Agent, the more the output depends on factors outside the Agent s contro, and the ess accurate the measurement of outputs, the ess intense the optima incentive structure 5. As a resut of these insights, the iterature identifies certain design features and considerations which enhance the effectiveness of a system of sanctions and rewards: Understand the strategy and deivery chain. Invove stakehoders deveop mechanisms in coaboration with the Agents to ensure consensus and appropriateness. Understand and consider the motivations of Agents. Ensure Agents can infuence the outcomes which are being incentivised. Ensure sanctions or rewards are of sufficient vaue or consequence to incentivise the desired behaviour. Introduce safeguards to protect against unintended behaviour Each of these features is discussed beow in reation to sanctions and rewards, though these essons are of wider reevance in designing a system of incentives, incuding the performance evers. Understand the strategy and deivery chain 3.19 To design an effective system of sanctions or rewards, it is first necessary to understand the context for their use. This requires an understanding of the drivers and the motivations that infuence performance against the objective (in the form of a strategy map) as we as an understanding of the various stakehoders and the performance evers that are empoyed to infuence them (in the form of a deivery map). Exampes of these maps, aong with some guidance, are avaiabe in Annex 3. The use of sanctions and rewards in the pubic sector 25

28 PART three The effectiveness of forma sanction and reward mechanisms 6. Communities and Loca Government (2006). Invoving stakehoders 3.20 Invoving the Agents to whom the sanction or reward mechanism wi be appied buids engagement and consensus. It can hep carify any potentia trade-offs between mutipe Principas and objectives, and hep the Principa better understand the motivations of the Agents Giving Agents some degree of contro of the sanction and reward mechanisms aso ensures that targets are achievabe and reevant ocay. The US Government Accountabiity Office concuded this was an important factor in the success of a vocationa and technica education programme, where individua states were given responsibiity for deveoping their own performance measures (GAO, 2006) Simiary, the UK government has recognised the importance of invoving Agents in the deveopment of incentive mechanisms. In a report on devoving government, giving oca organisations the authority and freedom to create oca incentives has been identified as key to driving improvements (HM Treasury, 2004a). The invovement of Loca Authorities has been a key feature in the setting of Loca Area Agreements, which were introduced as a resut of the Loca Government White Paper 6, and is especiay reevant given the ink to financia rewards in the form of Performance Reward Grants. The motivation of Agents 3.23 It is difficut to find objective evidence on motivation, but there are studies which show that introducing financia sanctions and rewards can have unintended consequences when at east some Agents have atruistic or other non-financia motivations An experiment at a day care centre in Israe found that the introduction of a sma financia penaty for parents arriving ate to coect their chidren actuay increased the number of ate arrivas (Levitt and Dubner, 2005). Before the fee was introduced, the ony penaty was the socia or mora one associated with making the day care providers wait onger. By aowing parents the possibiity of paying for ate coection, the daycentres reduced the perceived eement of socia disapprova (Seabright, 2004). This shows that where intrinsic or nonfinancia motivations dominate such as socia or mora conscience the introduction of financia incentives may undermine the vaue of the rewarded task, and have a counterproductive effect. We note, however, that the unintentiona effect of the sanction in this exampe may not necessariy be a bad one. Parents are wiing to pay for the additiona time their chidren are being supervised, and the centre is rewarded with additiona money to pay for it. If the sanction is set at an appropriate eve, the net effect coud therefore be an overa increase in utiity derived from the day care centre Sanctions or rewards can hep to focus Agents on tasks or outputs that woud have been negected if intrinsic motivation aone had been reied upon (Prentice et a, 2007). Where intrinsic motivation is high, operationa or reputationa sanctions and rewards may be more effective. In particuar, reputationa mechanisms which recognise good achievement, or highight poor performance, may act to strengthen intrinsic motivation. For exampe, severa stakehoders responsibe for schoo performance argued strongy that teachers are primariy motivated by an intrinsic desire to teach effectivey. In this area, Ofsted pointed to the motivationa vaue of their events for the head teachers of schoos judged to be outstanding, which aow peer recognition of good performance, and possiby assist career progression Where there is a mixture of potentia motivations, incentive schemes shoud be designed to aow for the possibiity of both types of motivation being present. For exampe, researchers in Sweden found that introducing financia rewards for donating bood reduced the overa number of donors (Gneezy, 2003) this effect is referred to as crowding out, as the financia rewards dispace 26 The use of sanctions and rewards in the pubic sector

29 THe effectiveness of forma sanction and reward mechanisms PART three the socia and mora motivation. The researchers found that by aowing individuas to give the payment to charity, the crowding out effect was fuy counteracted. This incentive scheme attracted those motivated by sef-interest through the monetary reward whie simutaneousy not deterring those who woud have donated bood anyway out of intrinsic motivation. In the UK pubic services context, Le Grand (1995) provides the exampe of the GP fund-hoders scheme, where GPs were aowed to keep any surpus on their funds, but ony use this surpus for a purpose beneficia to patients. He argues that this scheme aowed for the motivations of (what he terms) knights and knaves, in that the surpus can be used to benefit both the GPs and the pubic for instance, through improving premises or empoying more staff, thus easing work pressure and providing a better or more reaxed service. Infuence over the desired outcome 3.27 A sanction or reward mechanism shoud incentivise outcomes which can be sufficienty infuenced by the Agent s actions. Uness the Agent can expect their behaviour to affect the outcome, the mechanism provides itte motivation to strive for the desired eves of performance. This means that, as far as possibe, the sanction or reward mechanism shoud strip out aspects which cannot be infuenced by the Agent A good exampe of this is the Contextuaised Vaue Added measure of schoo performance in Engand. By adjusting for prior attainment, this measure heps to account for factors beyond the schoo s contro, and is generay seen as a better performance measure than the previous gross output measure which did not make any kind of adjustment. This approach ensures that even those Agents starting from the owest base are abe to infuence measured outcomes. For exampe, threshod schemes, which target absoute eves of performance and do not take past performance into account, may not reward Agents who improve the most as resut of starting from a ower base. Aowing Agents to be measured against their own past performance can hep to resove this issue, so ong as it does not unfairy penaise those that performed we in the past If the infuence of the Agent on the outcome cannot be satisfactoriy isoated, then a owerintensity sanction or reward mechanism may be more appropriate. High-intensity sanction or reward mechanisms which ink a arge proportion of pay, reputation or operationa freedoms to performance can impose too much risk on Agents if fina outcomes are beyond their contro (Grout and Stevens, 2003) If an Agent has imited infuence over outcomes, other aspects of performance over which the Agent has more contro coud be incentivised. For exampe, the Department for Environment Food and Rura Affairs and the Department for Transport have a shared objective to improve air quaity. Loca Authorities wi have an infuence over concentrations of air poutants in their area, but these may aso be impacted by factors which can ony be infuenced at a nationa or even internationa eve such as engine standards, and background emissions from aviation and shipping. Interviews with Loca Authorities suggested that incentivising particuar actions which are known to have a worthwhie, if minor, impact on air quaity such as awareness campaigns and car poo schemes may be more effective than trying to design a sanction or reward mechanism which targets absoute eves of air poutants. However, uness the strategy map (Figure 1) inking those outputs to the desired outcome is we understood, the overa effect coud be perverse e.g. setting up ots of car pooing schemes that do not get used. This approach shoud therefore be avoided where inks between outputs and outcomes cannot be modeed. If they are used, they may need to be accompanied by activity or quaity criteria which act to imit potentia distortions e.g. to avoid too many awareness campaigns that start to have diminishing returns. The use of sanctions and rewards in the pubic sector 27

30 PART three The effectiveness of forma sanction and reward mechanisms Seect the right performance measure 3.31 The performance measures that are used to sanction or reward performance are important. The wrong choice of metric can ead to Agents focussing their behaviour on some areas to the detriment of others. Where there are severa aspects to overa objectives, a successfu sanction or reward mechanism shoud capture a significant aspects of these objectives. It is ikey that a range of measures, rather than a singe performance measure, wi be needed. At the same time, care must be taken to keep the tota number of performance measures in a sanction or reward scheme manageabe Often the desired outcome is a ong term one, for exampe, that education or training programmes improve ifetime empoyabiity. This conficts with the need for performance measures to be avaiabe quicky for incentivising and accountabiity purposes. This tension can ead to perverse incentives. An exampe from the US concerns the administrators of a job training programme, who were judged on the empoyment status of participants soon after they competed the scheme. This ed to a situation whereby the scheme administrators were incentivised to discourage participants from seeking further training, regardess of the benefit to the participants ong-term goas, and to move straight into empoyment (Heckman et a, 2002). Therefore, if short-term outcomes are measured, then these shoud be demonstrated (empiricay or through expert opinion) as having a cear correation with the desired ong-term outcomes. An exampe of this approach is the Department of Heath and the Nationa Treatment Agency incentivising Drug Action Teams to increase the number of users in treatment programmes through inking a proportion of their funding to activity data. In this case, shortterm outputs are measured (numbers of peope engaged in drug treatment), because outcomes in drug use are ong-term, often taking years to emerge. However, there is research evidence that demonstrates a correation between the activity measured and the outcome (peope managing their addiction), meaning that activity data can be used as an acceptabe proxy The iterature aso contains exampes of cases where the choice of outcome measure has ed to gaming or strategic behaviour by Agents, such as anecdota evidence of cream-skimming by schoos to seect the best students. Threshod measures measures concerned with achieving an absoute eve of performance, such as the number of pupis achieving GCSE grade C or higher are seen as particuary vunerabe to gaming. This may resut in Agents concentrating their effort on those organisations or individuas who are performing just beow the threshod, to the detriment of the very good or very poor performers. In some cases this may be addressed by the natura incination of workers in the pubic sector to hep the most disadvantaged. However, where natura incination is not enough, there is the option of directy incentivising Agents to specificay prevent widening inequaities in society: The Government has recognised this risk and during the 2000 Spending Review introduced targets within Pubic Service Agreements (PSAs) that directy address inequaities. These targets make expicit reference to the outcomes to be achieved for ess we-performing groups as we as the rest of the popuation. (NAO, 2007) If outputs cannot be easiy specified, or if there are mutipe objectives or tasks where some are more easiy measured than others, it is ikey that a highintensity sanction or reward mechanism wi not be appropriate (Prentice et a, 2007). In this case, such a scheme is ikey to give rise to ineffective incentives, or to encourage the misaocation of effort by Agents. Rather, ower intensity mechanisms shoud be used, such as operationa or reputationa schemes, or financia schemes where ony a imited proportion of compensation is tied to performance (Burgess et a, 2005) Where outcomes are not easiy measurabe, or where there is a danger of unmeasured outcomes being negected, a sanction or reward mechanism which combines objective performance measures with subjective reviews may provide a soution. An exampe of this is the incentive scheme for schoos in Engand, which combines test resuts with a broad 28 The use of sanctions and rewards in the pubic sector

31 THe effectiveness of forma sanction and reward mechanisms PART three (but inevitaby more subjective) Ofsted assessment of the schoo s overa performance. Simiary, the Department of Heath and the Nationa Treatment Agency incentivise Drug Action Teams to increase the number of users in drug treatment programmes. They do this through tying centra funding to activity data as discussed above but have suppemented this with other activities. These incude Thematic Reviews, which are conducted in partnership with the Heathcare Commission, and ook at other aspects of drug treatment to ensure that this measure is not focussed on at the expense of others. Motivate a stakehoders 3.36 There is a risk that some performance evers and sanction or reward mechanisms ony motivate a sma proportion of the stakehoders namey, the best and the worst performing. In 2000 the Government recognised that Pubic Service Agreements which targeted ony aggregate performance figures coud be achieved simpy by focussing on a few stakehoders, whie the rest remained either static or even deteriorated. In the 2000 Spending Review it therefore set targets that directy addressed the particuar issue of preventing inequaities among the target popuation, and raising the standards of a. For exampe, the Department of Heath set a target to: reduce adut smoking rates to 21 per cent or ess by 2010, with a reduction in prevaence among routine and manua groups to 26 per cent or ess. However, these kinds of inequaities targets need to be set carefuy in order to have the appropriate motivating effect (NAO, 2007) Sanction and reward mechanisms run a simiar risk. If they ony reward the best performers and sanction the worst, then the majority of stakehoders may be content to maintain a mediocre eve of performance with no rea motivation to improve. In 2005 the Audit Commission sought to address this probem by introducing a direction of trave score for a Loca Authorities in its annua Comprehensive Performance Assessments. This meant that whie there was motivation to achieve a high absoute score, there was aso a motivation for a Loca Authorities to be improving their performance year-on-year. Team-based sanctions and rewards 3.38 Infuence over an outcome may be at the eve of the team rather than at the eve of the individua Agent. Makinson (2000) notes that teamwork better refects the way in which most pubic servants work, and measures of output are often ony avaiabe at the eve of the team in the pubic sector. For exampe, earning in a schoo is typicay a cumuative process rather than one which can be attributed to an individua teacher, and hence it may be more appropriate to reward a team, or the entire organisation, rather than a particuar individua. The Government Accountabiity Office has concuded that schemes offering significant financia incentives to a staff members in some US schoos incuding maintenance staff and bus drivers have contributed to significant performance improvements. This is party attributed to the camaraderie and coaboration fostered by the scheme among the empoyees However, the key risk of team-based mechanisms is the issue of free-riding. The arger the team, the more serious the free-riding probem, as it becomes increasingy easy for an individua to disguise their ack of contribution. But free-riding can be mitigated by peer pressure if team members can observe each other s effort. This means that the smaer the team, the more ikey team members are to be abe to monitor each other and reduce free-riding. Burgess et a (2005) found evidence to support this when they assessed some incentive schemes that were being pioted at HM Customs & Excise and Jobcentre Pus. Where this is not possibe or sufficient, free-riding can aso be mitigated through incentive schemes which combine both team and individua performance, as they require teamwork, as we as vauabe individua contribution. The use of sanctions and rewards in the pubic sector 29

32 PART three The effectiveness of forma sanction and reward mechanisms 3.40 The use of team-based measures may aso hep to overcome the difficuties of retaining intrinsic motivation within a sanction or reward mechanism, discussed in the previous section. A team-based sanction or reward scheme can maintain the coaborative, coective ethos of a pubic sector organisation without too great a focus on potentiay divisive, individua financia gains or achievements. Furthermore, Makinson (2000) notes that it is easier to integrate expicity financia sanction or reward mechanisms with non-financia incentives in a team framework. For exampe, having an office of the month is ess divisive than an empoyee of the month, yet the former can sti be rewarded by a shared financia bonus. Set sanctions and rewards at a cost-effective vaue 3.41 The aim of a sanction or reward mechanism is to obtain a cost-effective increase in performance. In other words, the cost of the reward for additiona units of performance must be proportionate to the vaue of the additiona units of performance both the vaue to the Principa, and the increased effort required by the Agent. If there is a disconnect between the costs and benefits in a sanction or reward mechanism, it wi not motivate the intended additiona effort, and wi ead to the inefficient use of resources Whether the mechanism is financia, operationa or reputationa, the vaue or consequences of the sanctions or rewards need to be sufficienty high to motivate the desired behaviour. Indeed, the Government Accountabiity Office concuded that inappropriatey vaued incentives appear to be one of the main reasons sanction or reward mechanisms fai (GAO, 2006) Through our case studies, we found severa exampes where the size of the sanction or reward was widey seen to be insufficient to have the desired effect: Tariffs paid to farmers to provide environmenta goods, such as retaining hedgerows, under the Rura Deveopment Programme for Engand have been overtaken by rising commodity prices. This rise has ed to a drop in the uptake of environmenta contracts, and an expectation that a number of existing contracts wi not be renewed. The Vehice Excise Duty concession avaiabe to Heavy Goods Vehice (HGV) operators in possession of a Reduced Poution Certificate is much smaer than the capita investment required to achieve the necessary emission standards. This has ed to take-up of the scheme being ow, with ony around 15 per cent of HGVs in possession of a certificate. Various UK pubic sector performance-reated pay schemes, reviewed by Prentice et a (2007), were deemed too sma. For exampe, awards of 50 to 150 paid in the Defence Aviation Repair Agency, and a bonus of 400 paid to a staff awarded an exceeding performance rating in the former Department for Constitutiona Affairs. Makinson (2000) recommended that performance-reated pay bonuses shoud represent at east five per cent of base saary in order to be effective By way of contrast, the reative success of a job training partnership programme in the US is attributed to the size of the rewards, whereby the programme awarded oca training centres as much as a 20 to 30 per cent increase in their reguar budgets if they successfuy trained or paced customers. These awards were particuary vaued as training centres coud use them more fexiby than reguar budget aocations (Heckman et a, 2002) As we as rewards being of adequate vaue, sanctions shoud correspondingy be stringent enough to incentivise behaviour. Propper et a (2007) discuss the hospita waiting time target scheme in Engand, which used the dismissa of key managers of hospitas as a sanction for poor performance. In comparison to Scotand which had a simiar focus on reducing hospita waiting times, but without the high-intensity sanction regime there is evidence that the Engish scheme significanty reduced waiting times. 30 The use of sanctions and rewards in the pubic sector

33 THe effectiveness of forma sanction and reward mechanisms PART three 3.46 Affordabiity constraints can imit the size of rewards, and arge financia sanctions for beow average performers may not be poiticay acceptabe such as cutting the funding to a poory performing hospita. The foowing exampes, from both the iterature and our own case studies, show why modeing the costs and benefits of an incentive scheme can hep to determine the cost-effective eve of a reward or sanction: Exampe 1 Department for Environment, Food and Rura Affairs Rura Deveopment The Department s Service Leve Agreements (SLAs) with its main deivery bodies (the Forestry Commission, Natura Engand and the nine Regiona Deveopment Agencies) incuded itte scope for it to sanction poor performance with respect to rura deveopment. Most of the funding fows from the European Commission, and is subject to their reguations, which if not satisfied raises the possibiity of disaowance. The current SLA with the Regiona Deveopment Agencies incudes a imited sanction, whereby the Agencies are exposed to five per cent of the disaowance risk if they fai to deiver. However, the funding received from the Rura Deveopment Programme for Engand is sma in comparison to the Agencies other funding streams, and so the Programme is unikey to be a high priority. The Department is ooking to increase the risk percentage shared with the Agencies. In schoos in Engand and Waes, a new upper pay scae for teachers reated to performance was introduced in 1998, but abandoned six years ater as it became apparent that the scheme was unaffordabe. This was due to a fawed scheme design, which initiay brought around a third of teachers into the highest pay scae, but over time brought in far more individuas than expected. This eventuay made the scheme financiay unreaistic, and resuted in the two top scaes being taken away. In the introduction of the GP contract, the NAO (2008) found that a ack of understanding of the costs and benefits resuted in a arge overspend and a decrease in overa productivity. The new contract for genera practice contributed to improved recruitment and retention of GPs, with numbers increasing from around 27,000 to neary 31,000 since However, the contract cost the Department 1.76 biion more than it had originay budgeted. The NAO found that in the first two years of the contract, productivity had faen by an average of 2.5 per cent per year. Whie the number of consutations with patients had increased, these were not in proportion with the increase in costs. The argest overspend of the contract was due to an underestimation of the amount that GPs woud earn from the pay for performance scheme, known as the Quaity Outcomes Framework (QOF). In addition, the inks to fina outcomes were not proven; whie there was evidence that the QOF has improved consistency in the quaity of care, it is too eary to say if overa patients heath has improved as a resut (NAO, 2008). NAO (2001a) highights the exampe of HM Customs & Excise, which deveoped a mode to estimate the current and future eves of smugging and assess the impact of HM Custom & Excise activities incuding the sanctions and rewards on the eve of smugging. This heps the Department to make resource decisions, aowing it to understand the benefits of achieving particuar eves of performance, and so determining the appropriate eve of a sanction or reward that shoud be associated with a particuar eve of performance. The Department for Transport carried out a cost-benefit anaysis to provide an indication of the size of any concessionary eve of Vehice Excise Duty to Heavy Goods Vehice operators that woud be necessary in order to incentivise the uptake of ow emissions Euro V vehices. This anaysis baanced the expected costs of the scheme against monetised benefits to heath and the environment to determine if the scheme was cost-effective. The use of sanctions and rewards in the pubic sector 31

34 PART three The effectiveness of forma sanction and reward mechanisms 7. Ambuance times misreported. The Guardian. 15 August A&E wards accused of fidding waiting times to hit targets. Daiy Teegraph. 6 October Introduce safeguards against unintended behaviour 3.47 Experience has shown that, particuary for high-intensity sanction and reward mechanisms, undesirabe manipuation of the system may occur. Exampes incude waiting ist targets in the heath service, where NHS Trusts manipuated data to show better resuts than in fact had been achieved (NAO, 2001b). In the case of response time targets, aso in the heath sector, authorities expoited ax definitions in the targets to measure response in partia ways 7, or atered their procedures so that they coud egitimatey caim that response had started earier in the process Severa mechanisms have been used to counter such behaviour, incuding the vaidation or audit of underying data systems, independent assessment of performance achieved, and tighter specification of what constitutes performance. Academics have aso suggested introducing a degree of uncertainty over the quantum of, or precise basis for, sanctions and rewards, to prevent over-cacuated management responses However, the existence of incentives whether they are performance evers or expicit sanctions or rewards may not necessariy ead to dysfunctiona behaviour. Keman and Friedman (2007) suggest three circumstances in which dysfunctiona behaviours do not arise or are at east imited: When additiona effort in targeted activities may compement rather than substitute for other desired, but untargeted, activities. For exampe, in order to improve their performance in admitting patients from Accident & Emergency, hospitas reengineered processes that aso improved the performance in other areas. Sef-imitation, whereby the dysfunctiona response provokes a countermanding action. For exampe, unnecessary admissions to inpatient hospita wards from Accident & Emergency create a burden for managers of inpatient wards, who then object to those types of admissions. When management take actions to imit dysfunctiona behaviours, such as adding new dimensions to a target, or by cutivating the pubic sector ethos among empoyees. For exampe, having a target to reduce eective surgery waiting times at the same time as having a target to reduce Accident & Emergency waiting times may have heped to imit hospitas from substituting their efforts from eective surgery to Accident & Emergency. These issues need to be considered at the design stage of a programme, and responses seected and then reviewed after initia operation. Concusions The Principa needs to understand the strategy for achieving the objective as we as the deivery chain that wi hep to achieve it before considering the use of performance evers and the accompanying sanctions and rewards. The sanction and reward mechanism shoud be deveoped together with the Agents so as to ensure consensus and appropriateness. The sanction and reward mechanism shoud consider the motivations of Agents. The Agents must be abe to infuence the outcome which is being incentivised. The incentivised outcome or intermediate output must ead to the overa desired outcome from an organisation. A stakehoders shoud be motivated to improve. The sanction and reward mechanism shoud take into account the team structures of the organisation. The sanctions or rewards must be of sufficient vaue or consequence to incentivise the desired behaviour. If there is a risk that sanctions or rewards coud ead to unintended behaviour, mechanisms must be put in pace to minimise this. 32 The use of sanctions and rewards in the pubic sector

35 THe effectiveness of forma sanction and reward mechanisms PART three Measure 3.50 This stage of the process is ony reevant to sanction and reward mechanisms that are inked to measured eves of performance. Sanction and reward mechanisms can be categorised according to whether they are inked simpy to compiance, or whether they are inked to measured performance. For exampe, concessions from Vehice Excise Duty are offered to vehices that hod a Reduced Poution Certificate. This reward mechanism reies on the process of assessing whether a vehice fufis the requirements for the certificate a simpe yes or no decision. However, the aocation of funding to Drug Action Teams through Pooed Treatment Budgets needs more compex data systems that capture activity data in drug treatment centres throughout the country. Deveopment of systems to coect performance data 3.51 Once the performance measures to be incentivised have been chosen, it is necessary to ensure adequate systems are in pace to coect data on the chosen measures. Data systems shoud ensure that data are we-defined, timey and reiabe 9. The NAO has reguary found that ess than 50 per cent of the data systems underpinning Pubic Service Agreements to be fit for purpose (NAO, ) The Government Accountabiity Office (2006) points to the importance of efforts by the Office of Management and Budget (the body responsibe for improving the performance of federa programmes) to improve agencies abiities to deveop high quaity, resuts-based performance measures as a preude to introducing incentive schemes. Simiary, the UK government has identified accurate and timey pubication of performance data as essentia to strengthening incentives through sanction or reward mechanisms (HM Treasury, 2004a). Exampe 2 Department of Heath and Nationa Treatment Agency treatment for drug misuse The Nationa Treatment Agency is responsibe for increasing the number of probem drug users in effective treatment programmes. As a resut of substantia investment into data systems, the Agency has access to robust and extensive data which it can use to aocate funding on a fairer basis. Using a common, trusted data system has been critica in getting buy-in from stakehoders for this new funding mechanism, which rewards those who are using ess centra funding per individua treated with more money, and sanctions the reativey inefficient areas with ess. Athough there are moves towards tying eements of the funding to the eves of drug use in an area rather than the voume of treatment activity the quaity of data at a oca eve is not good enough yet. Anaysing and reporting the performance data 3.53 Once the coected data have been vaidated, they need to be anaysed, interpreted and then reported in a cear and concise way so that a stakehoders are abe to understand performance. Our survey showed that 94 per cent of respondents beieved that their anaysis of performance data was robust enough to make reiabe decisions when appying sanctions or rewards. However, other work undertaken by the NAO on the quaity of data systems underpinning major government programmes has found significant weaknesses (NAO, ) For New York s Compstat programme which aows detaied crime patterns to be anaysed and crime reduction pans to be deveoped a 25 person team is responsibe for centra anaysis and quaity assurance of the data coected from each oca precinct. This then paves the way for the weeky accountabiity meeting (HM Treasury, 2004a). Concusions Data systems shoud ensure that data are wedefined, timey and reiabe. 9. Choosing the right FABRIC, A Framework for Performance Information. HM Treasury, Cabinet Office, Nationa Audit Office, Audit Commission, Office for Nationa Statistics. Data systems need to be supported by a reiabe management structure that anayses and reviews the data. The use of sanctions and rewards in the pubic sector 33

36 PART three The effectiveness of forma sanction and reward mechanisms Appy 3.55 For a system of sanctions or rewards to have the desired incentivising effect, it must be appied in a timey, consistent, and transparent manner The Department for Communities and Loca Government tod us that as of 2008 it is using a Housing and Panning Deivery Grant to incentivise Loca Authorities to provide housing and to put oca pans in pace. However, the housing eement is based on average deivery over the ast three years, which means that current actions to increase housing suppy wi ony graduay infuence grant aocations, which may weaken the incentivising effect If rewards or sanctions are not awarded as expected, Agents earn that their additiona efforts are not worth the cost or risk. The Government Accountabiity Office (2006) notes that if rewards are paid indiscriminatey, or sanctions are not evied as expected, Agents can earn that no additiona effort is required to benefit. Figure 10 shows that over 40 per cent of deivery chains that responded to this question acknowedged that they are not aways appied consistenty, with neary a quarter not even knowing whether they were or not. 10 Are sanctions and rewards appied consistenty? Unsure 24% Aways 35% Source: Nationa Audit Office survey of Departments Mosty 41 % 3.58 Consistency in appication is one of the principes highighted by Macrory (2006) in his report on effective reguatory sanctions. He notes that faiure by reguators to foow up ow-eve enforcement actions such as warning etters with the threatened sanction, means that they wi not be taken seriousy and crediby by firms. Consistency is particuary important in the pubic sector, where Agents often have more than one Principa trying to infuence them. Given the many, and potentiay conficting, objectives hed by different Principas, it is possibe that Agents may pay off one against another (Marsden and French, 2002). Issues with mutipe Principas can be mitigated by ensuring that eves of performance which trigger a sanction or reward are ceary defined and understood, and are consistenty appied without exception Sanction or reward mechanisms need to be transparent, not ony for those to whom they wi be appied, but for the motivation of other stakehoders. If an Agent does not know why they were sanctioned or rewarded for something, then the motivationa vaue of the sanction or reward is diminished, as the Agent is not cear what is expected of them. As far as other stakehoders are concerned, if they cannot see how sanctions or rewards are appied to others, they may not fee motivated to behave accordingy. Senior staff at the Dorset and Somerset Strategic Heath Authority beieve that transparency is one of the key characteristics that made their incentive scheme successfu: The rewards and sanctions structure is not a private bargain between management teams, in the manner of some NHS contracting or commissioning. Instead, the system is highy visibe externay. That visibiity goes beyond organisationa boundaries and sends a message to frontine staff. Once given to a Trust, staff wi reasonaby expect to see how the money is spent. The fact that the money fows throughout a financia year makes it simper for Trusts to pan investments arising from their success. Cruciay that panning is in the knowedge that reward is guaranteed if they meet their targets. As a resut, the ink between achievement and benefit is both immediate and transparent. (HM Treasury, 2004a). 34 The use of sanctions and rewards in the pubic sector

37 THe effectiveness of forma sanction and reward mechanisms PART three Concusions Sanctions and rewards need to be appied in a timey manner to have the desired motivating effect. Sanctions or rewards need to be appied fairy and consistenty. The process of appying sanctions or rewards needs to be transparent. Review 3.60 Once a system of sanctions or rewards is in pace, the Principa shoud evauate whether it is having the desired effect on performance. This is compicated by the many other factors that impact on performance, making it difficut to isoate the unique contribution of the sanctions or rewards themseves. However, particuary where there is significant risk to the success of the poicy or a arge amount of resource is invoved such evauation is essentia. Without it, faws in the design, measurement, and appication of the system may be causing unintentiona behaviour, eading to the faiure of the poicy and wasted resources Prendergast (2001) found that unmeasured outcomes in a Los Angees poice force sanction scheme ed to a focus on ony some aspects of desired outcomes. The scheme was designed to combat vioence against criminas during arrests, yet ed to a disproportionate fear of discipinary action in case of compaints from incorrecty arrested citizens. Given the trade-off between arresting the innocent and not arresting the guity, the scheme resuted in officers being strongy biased towards the atter. Here, the overy punitive system ed to the inefficient outcome of etting too many criminas free, or not arresting them in the first pace. This hindered the achievement of the institution s overa objective of fighting crime, as evidenced by the rapid increase in homicide rates after the introduction of the scheme In our survey, 60 per cent of the deivery chains that used expicit sanctions or rewards, and responded to this question, beieved that they had improved performance. However, interviews with the organisations invoved highighted itte evidence of the direct ink between the sanctions and rewards, and the eves of performance Reviews need to be both fit for purpose, and sufficienty reguar. With different types of government programmes, and different systems of sanctions and rewards, the way in which effectiveness is assessed needs to be taiored accordingy. Various approaches are avaiabe for assessing the effectiveness of sanction and reward mechanisms. Evauations can be grouped into the foowing types: Randomised trias or experiments. These are the most robust assessments, as they use a ceary defined and monitored contro group to understand what performance woud be without any intervention. This aows the causa reationships between the sanction or reward mechanism and the measured outcomes to be identified. However, this approach is expensive, time-consuming and can be poiticay difficut as some arbitrary recipients benefit from the programme whie others are excuded. As a resut, there are very few such studies. Natura experiments. In some cases, natura experiments can provide a good aternative to randomised trias. One exampe in a UK pubic sector context is the comparison of schemes regarding hospita waiting time targets in Engand and Scotand, where the Engish imposed a highintensity sanction regime whie Scotand did not (Propper et a, 2007). Because of the simiarity of the two systems, many of the other determinants of outcomes besides the sanctions were not reevant to changes in performance, and so differences coud be attributed, within reason, to the sanctions themseves. Econometric or statistica anaysis. This can be used to compare performance before and after the introduction of a sanction or reward mechanism, as we as to compare two simiar popuations, one of which does, and one which does not use a mechanism. It makes the comparison by statisticay removing the factors which may affect the outcome, and which vary over time The use of sanctions and rewards in the pubic sector 35

38 PART three The effectiveness of forma sanction and reward mechanisms or between popuations such as the eve of funding or the demographics of the popuation. The robustness of these kinds of studies depends criticay on the quaity and quantity of data, but even when the data is good, causation is much harder to determine than correation. Hypothesising. Data on outcomes, and on potentia expanatory variabes other than the sanction or reward mechanism, is often difficut, expensive, or sow to obtain. In these cases, the assessment of effectiveness is imited to examining the path of output measures over time, and hypothesising inks with the introduction of (or refinements to) sanction and reward mechanisms. Net output measures, which attempt to measure the vaue added by the programme, are preferabe to gross or raw output measures. One exampe of a net output measure is the Contextuaised Vaue Added measure of pupi test performance used in Engish schoo eague tabes since This measures the change in exam performance for each pupi between key stage assessments in an attempt to get at the vaue added by the schoo. This has been introduced aongside the gross output measure of absoute number of GCSE grades, which coud be infuenced by many factors outside the schoo s contro. Quaitative evauation. Finay, in some cases the effectiveness of a sanction or reward mechanism is assessed on a purey quaitative basis for exampe through questionnaires and/or structured interviews. This approach is inevitaby subjective, but can provide insights to compement more quantitative approaches Nevertheess, where feasibe, scope shoud be buit in to revise and update the sanction or reward mechanism in the ight of the best avaiabe evidence on its effectiveness. For exampe, in a vocationa and technica education programme (GAO, 2006), officias have attributed the programme s success to the abiity of American states to periodicay revise their target and measures during annua negotiations of their state pans In addition to the risks of unintended consequences emerging over time, technoogy may be introduced that aters performance expectations, or the priorities of an organisation may change (GAO, 2006). Certain performance standards may aso become the norm and may no onger need to be incentivised. These are a reasons for why such mechanisms need to be continuay reviewed. Exampe 3 Department for Communities and Loca Government Panning Deveopment The Department inked the financia reward of its Panning Deveopment Grant (PDG) to the speed with which Loca Authorities processed panning appications. This ed to significant improvements in performance over the five years PDG was paid. However, by the end, performance appeared to have reached a reativey steady state. Now that the cuture of oca panning has changed with regards to timeiness, the Department is now introducing a revised system of financia rewards which are inked to the number of houses constructed and putting oca pans in pace. There wi sti be some incentive to maintain performance on panning appications because some grant wi be abated from Loca Authorities who do not meet targets for timeiness As these methods show, robust assessment of effectiveness is difficut. Reviews of the academic iterature find ony a sma amount of empirica evidence as to whether these mechanisms actuay achieve the goas of the programme or Department (Burgess et a, 2002). Prentice et a (2007) found ony seven empirica studies on UK pubic sector financia incentive programmes after Concusions Frequent reviews of the sanction or reward mechanism are required in order to assess its effectiveness, amend it for any design faws, and take into account changing needs over time. 36 The use of sanctions and rewards in the pubic sector

39 THe effectiveness of forma sanction and reward mechanisms PART three The most accurate assessment of the effectiveness of a sanction or reward mechanism woud incude an estimate of what woud have happened in the absence of one. Estimates of what woud have happened are technicay difficut, require a ot of data, and can take a ong time. Methodoogies incude the use of contro groups and the econometric anaysis of comprehensive data sets. The difficuty and expense of conducting an estimate of what woud have happened means that the effectiveness of a sanction or reward mechanism is generay assessed in terms of short-term measured outputs based on administrative data. At the simpest eve, gross outputs are a measure of raw outputs of a programme, and have the advantage of being easy to coect and understand. If avaiabe, net outcome measures can be more usefu in measuring the contribution of a programme to the outcome. These measures seek to estimate the vaue that is added by the programme. In the absence of other data, the effectiveness of a sanction or reward mechanism can be assessed on a quaitative basis, for exampe through canvassing the opinions of participants in the scheme. and confidence in the quaity and reiabiity of the coected data to be buit up. In particuar, there is a strong case for a mechanism to be initiay impemented on a tria or piot basis (Armstrong and Brown, 2001). This aows its impact to be accuratey assessed, as it can be compared to a counterfactua Furthermore, phasing in a sanction or reward mechanism over time aows Agents the opportunity to deveop the necessary knowedge about the mechanism, and the abiity to impement it. This is critica to the success of the scheme. For exampe, the US Air Force found that the most important factor in impementing its successfu performancebased contracting programme was empoyee training that focussed on how the performance mechanism worked (GAO, 2006). Concusions Phased impementation aows the design of a sanction or reward mechanism to be tested. It aso aows data systems to be deveoped, and those affected to deveop the necessary knowedge about the mechanism and the abiity to impement it. There is a strong case for initiay impementing a sanction or reward mechanism on a piot basis. Impement 3.67 Aowing a sanction or reward mechanism to be phased in over a period of time aows a panned and incrementa change which is important to the scheme s success. Indeed, Armstrong and Muris (1998) argue that a hasty impementation is often responsibe for a scheme s faiure, rather than any fauts with the scheme itsef A phased impementation aows organisations to ensure that the scheme is effectivey designed. There are often unintended consequences associated with a sanction or reward mechanism, which means that aowing sufficient time to test the mechanism is vita before tying it to sanctions or rewards. A phased impementation aso aows time for data coection systems to be deveoped, The use of sanctions and rewards in the pubic sector 37

40 PART ONE Annex ONE Methodoogy Methodoogy In order to understand and to scope the key issues for this study, we reviewed various academic iterature, and discussed the topic with academics who have worked in the area of incentives. We empoyed Deoitte to conduct the iterature review, which covered both the theory and the practica appications of sanctions and rewards, in both the pubic and private sectors, in the UK and overseas. We then conducted a two-stage anaysis process: 1 We undertook a survey of the poicy eads for significant government programmes, asking questions on the use of sanctions and rewards in the deivery chains underpinning poicy deivery. This provided us with an overview of the different types of sanctions and rewards being used, and the different situations in which they were appied. We pioted the survey with the Department of Heath. 2 We then seected eight major programmes as case studies. Seection was primariy based on obtaining a mix of deivery chains that did, and did not, use any type of sanction or reward mechanisms. We aso tried to get a mix of deivery chain types. The eight we seected were: Programme Department Reported using sanctions Achieved by or rewards? Homeessness Communities and Loca Government No Service deivery Panning deveopment Communities and Loca Government Yes Service deivery Educationa attainment Chidren, Schoos and Famiies Yes Service deivery Rura productivity Environment, Food and Rura Affairs No Infuencing Air quaity Environment, Food and Rura Affairs No Infuencing Air quaity Transport Yes Infuencing Drug harm Home Office No Infuencing Drug harm Heath Yes Infuencing 38 The use of sanctions and rewards in the pubic sector

41 Methodoogy Annex ONE 3 For the seected programmes we conducted interviews with the Departmenta poicy eads and key deivery stakehoders to gain a more detaied understanding of the deivery chain. We asked about the use of evers within the deivery chains, views on effectiveness of deivery, and opportunities to enhance effectiveness. Where sanctions or rewards were being used we asked for more detai about how they worked in practice. Survey We surveyed a the major programmes of Government as defined by the 2004 Spending Review Pubic Service Agreement targets in operation at the time of the survey. We achieved a 63 per cent response rate. A review of the respondents and non-respondents did not impy a significant bias to the resuts; the programmes that responded showed a good mix of Departments and types of deivery chain. A number of responses were not comprehensive in describing a sanction or reward mechanisms used in the deivery chain. Survey respondents were ony asked to outine forma sanction or reward mechanisms. In the case studies we found a number of additiona sanction or reward mechanisms which had not been incuded in survey responses. The use of sanctions and rewards in the pubic sector 39

42 annex PART two ONE Different types of sanctions or rewards Different types of sanctions or rewards Type Organisationa eve Team eve Individua eve End user eve Financia sanction or reward mechanisms Contract/grant renewa One year contract renewa for good performance specified in the Department for Chidren Schoos and Famiies contract with Capita. Contract/payment hodback The renewa of Air Quaity Grants to Loca Authorities is dependent on achievement of the targets specified in the origina grant. Ten per cent contract hodback by Ontario Reaty Corporation for faiure by the faciities management company to meet performance standards. A Service Leve Agreement with the Regiona Deveopment Agencies for funding provided through the Rura Deveopment Programme. The Agencies are subject to five per cent of the disaowance risk for non-compiance with EU reguations. Deduction from benefit payments to out-of-work benefit recipients by the Department for Work and Pensions, in the event of noncompiance with mandatory activity. Payment withhed by the Department for Environment Food and Rura Affairs for non-compiance by farmers and andowners with the contract terms of the Rura Deveopment Programme. 40 The use of sanctions and rewards in the pubic sector

43 Different types of sanctions or rewards Annex two Type Organisationa eve Team eve Individua eve End user eve Financia sanction or reward mechanisms continued Bonus above estabished baseine Bonus funding above a standard amount through the Panning Deivery Grant to Loca Authorities, based on performance criteria reated to the speed of processing panning appications. Thirty per cent of the Department for Internationa Deveopment s funding to the Internationa Fund for Agricutura Deveopment made contingent upon performance system-wide reform is required. Increased funding by the Department for Internationa Deveopment provided to UNICEF or recipient nationa governments for good performance. Bonus given to US Job Training Partnership teams on the basis of outcomes, such as job pacements and earnings by trainees. Bonus of up to 7.5 per cent paid to teams in Jobcentre Pus on achievement of five performance targets. Gendae Schoo in the US provides financia incentives to a academic and support staff upon the schoo s achievement of a number of academic, invovement and satisfaction-reated performance targets. Sma performancereated bonuses in different UK government Departments such as the Home Office, Defence Aviation Repair Agency and the former Department for Constitutiona Affairs. Team-based performance pay schemes for civi servants in Braziian Tax Authority based on objective measures of the amount of tax coected, combined with individua financia rewards based on a subjective assessment. Financia incentives offered by the Department for Work and Pensions to recipients of incapacity benefits or one parent benefits, to hep them move into, and remain in work. Bonus/deduction from empoyees saaries based on the over or under-achievement of individua and team targets, by a Hong Kong train operator. Bonus based on stretch target Makinson performance pay incentive scheme in HM Customs & Excise, where teams had to achieve a basic target before being eigibe for a bonus based on achievement of a stretched target. The use of sanctions and rewards in the pubic sector 41

44 annex two Different types of sanctions or rewards Type Organisationa eve Team eve Individua eve End user eve Financia sanction or reward mechanisms continued One-off bonus Book vouchers and points, which can be converted into prizes, are presented to pupis by some UK schoos for educationa attainment and schoo attendance. Unit-based payment/penaty Fifty-five per cent of funding provided by the Nationa Treatment Agency to Drug Action Teams is based on activity data to incentivise the treatment of more peope. Piece-rate payment scheme introduced in a US windscreen instaation firm, in which workers pay was inked to the number of windscreens they instaed. A proportion of GPs income is reated to the quaity of their care, based on a siding scae of achievement through the UK Quaity and Outcomes Framework. Renewabe Transport Fue Obigation a reguatory requirement that a proportion of a road fue sod in the UK is from renewabe sources. Incentivised by fue duty discount per itre of renewabe fue sod and penaised for each itre not suppied. Levy of a NOx anding charge at Heathrow and Gatwick, on airines, by the Civi Aviation Authority. Share in savings UK GP fund-hoders scheme whereby GPs were aowed to keep any surpus on their funds but had to use this for any purpose that was beneficia to patients. Reduction in charge Lower rates of excise duty offered to Heavy Goods Vehice operators who achieve target eves of emissions and obtain the Reduced Poution Certificate. 42 The use of sanctions and rewards in the pubic sector

45 Different types of sanctions or rewards Annex two Type Organisationa eve Team eve Individua eve End user eve Reputationa sanction or reward mechanisms Pubic ranking UK schoo eague tabes. Pubication of reports by Ofsted, which assess and rank the performance of schoos. Monitoring of Jobcentre Pus district offices performance against targets, and comparison across districts through a eague tabe. Award Green Fag Award for good management of parks and open spaces, given by the Department for Communities and Loca Government to Park Management and Maintenance. Designating beacon status to Loca Authorities to recognise exceence in performance. Outstanding providers event for teachers hosted by Her Majesty s Chief Inspector and Ofsted. Pupis rewarded for academic performance through a number of incentive schemes, incuding etters to parents and Head Teacher commendations for good performance, as we as merit stamps in pupis diaries. Pubications Pubication of the environmenta poicies submitted to the Office of the Rai Reguator, by the Department for Transport. The scheme is appied to icence hoders, such as Network Rai, and Train Operating Companies. The use of sanctions and rewards in the pubic sector 43

46 annex two Different types of sanctions or rewards Type Organisationa eve Team eve Individua eve End user eve Operationa sanction or reward mechanisms Decreased oversight Decreased oversight and greater freedom from centra management given to UK hospitas who attain Foundation Trust status by meeting certain performance standards. Reduced or no inspection from Ofsted on the basis of a schoo s performance. Dismissa or discipinary action Shutting down schoos or putting them into specia measures on the basis of unsatisfactory performance gauged by Ofsted inspections. Discipinary action against Los Angees poice officers in case of citizen compaints. Dismissa of key managers of hospitas in the UK, on the basis of poor performance, measured in terms of waiting times for eective hospita procedures. Pupis sanctioned for misbehaviour through a demerit system. A pupi with a certain number of demerits is required to undergo supervised earning from a teacher. 44 The use of sanctions and rewards in the pubic sector

47 Guidance for deveoping and impementing sanction and reward mechanisms Annex three Guidance for deveoping and impementing sanction and reward mechanisms The purpose of this Annex is to provide some practica guidance for organisations that are considering using sanction or reward mechanisms to hep achieve their objectives. Figure 11 outines a standard process which can be foowed: 11 Process for designing and impementing a system of sanctions and rewards to achieve an objective (1) Carify the objective (2) Map the strategy (3) Map the deivery chain (4) Anayse the stakehoders Sanction or reward mechanism (5-10) Design (13) Review (11) Measure (12) Appy (14) Impement Source: Nationa Audit Office The use of sanctions and rewards in the pubic sector 45

48 Annex three Guidance for deveoping and impementing sanction and reward mechanisms Objective 1 Ceary define a measurabe objective. Map the strategy 2 Identify the activities and the outputs that are needed to achieve the objective (see the exampe Strategy Map in figure 1). [p. 7 and 47]. Map the deivery chain 3 Identify and engage with the key stakehoders, map the performance evers and incentives in use (see the exampe Deivery Map in figure 2). [p. 8 and 48]. 4 Use the deivery map to undertake a stakehoder anaysis. Identify where the main barriers are, and the evidence on the effectiveness of the current programme, the performance evers, and their associated sanction or reward mechanisms (see Stakehoder Anaysis tempate in Annex 3). [p. 49]. Designing sanction and reward mechanisms 5 Understand the fu range of sanction or reward mechanisms avaiabe financia, reputationa and organisationa (see Annex 2). [p. 40]. 6 Assess the motivations of the key payers. Assess the baance between financia and atruistic motivations, and whether they operate at organisationa, team or individua eve. [p. 26 and 40]. 10 Introduce safeguards to prevent unintended behaviours. Approaches can incude maintaining a degree of independence in performance assessment and vaidating key performance measures [p. 32]. Measure performance 11 Deveop and impement data systems that coect and anayse timey and reiabe performance data. [p. 33]. Appy the sanction or reward 12 Deveop the expertise to appy the sanctions or rewards in a timey, consistent and transparent manner, especiay when using compex contractua arrangements with deivery agents. [p. 34]. Review effectiveness 13 Buid reguar interna and externa reviews of the effectiveness of the sanction or reward mechanism into the programme s overa performance management cyce. [p. 35]. Impement 14 Where possibe, phase in new sanction or reward mechanisms graduay, or on a piot basis, to identify and address any dysfunctiona behaviour. [p. 37]. 7 Identify outcomes over which the payers have contro or significant infuence. [p. 27]. 8 Link the incentives to performance measures which ead to the desired (ong-term) outcomes in a predictabe way. A singe measure may not capture the reevant aspects of performance, but any set of measures must be kept manageabe. [p. 28]. 9 Ensure the rewards and sanctions are costeffective. Where they invove financia eements, it may be sensibe to mode the operation of the system to hep define appropriate vaues or ranges. [p. 30]. 46 The use of sanctions and rewards in the pubic sector

49 Guidance for deveoping and impementing sanction and reward mechanisms Annex three Strategy Map Figure 12 shows a generic strategy map for an objective. It shows how an organisation can take an objective, and identify the combination of activities and outputs that shoud hep to achieve it. It shoud be based on evidence, or at east consensus, on what drives the outcome. It can then be used to identify the stakehoders that wi deiver those outputs in the form of a deivery map. 12 Strategy Map Objective Intermediate outcome 1 Intermediate outcome 2 Intermediate outcome 3 Intermediate outcome 4 Output 1 Output 2 Output 3 Output 4 Output 5 Output 6 Activity 1 (Organisation 1) Activity 2 (Organisation 2) Activity 3 (Organisation 3) Strong infuence Moderate infuence Weak infuence Source: Nationa Audit Office The use of sanctions and rewards in the pubic sector 47

50 Annex three Guidance for deveoping and impementing sanction and reward mechanisms Deivery Map Figure 13 shows a generic deivery map for an objective which is trying to infuence the behaviour of a target popuation. It shows how an organisation can take an objective and its strategy, and map the reationships between the stakehoders that are invoved. It then shows the performance evers that are used to infuence those stakehoders, and any sanctions or rewards that are attached to them. Instructions Be cear on the scope of the deivery map. The nature of pubic services means they are ikey to overap with many other objectives. Whie it is important to understand a the key stakehoders and infuences, it is aso important to keep a deivery map focussed on the objective to avoid unnecessary compexity. Incude externa infuences, even though they cannot be managed. It is important to know what ese infuences a deivery chain so that the performance evers and incentives can be designed around them. Distinguish between strong infuences and weak infuences, as this wi hep to focus the evers and incentives on where they can have most impact. Invove as many of the stakehoders as possibe to get a fu understanding of the infuences and incentives. 13 Deivery Map Funding Performance Management Organisation 1 (Activity 1) Funding Pubished performance eague tabes Department Organisation 2 (Activity 2) Education Contract Organisation 3 (Activity 3) Education Target popuation Financia bonus Taxation Externa infuences Performance ever Sanction or reward Strong infuence Moderate infuence Weak infuence Source: Nationa Audit Office 48 The use of sanctions and rewards in the pubic sector

51 Guidance for deveoping and impementing sanction and reward mechanisms Annex three Stakehoder Anaysis The foowing framework is a tempate which can be used to anayse the issues faced by stakehoders within the deivery chain, and deveop ideas and recommendations for strengthening the current performance evers with the hep of sanctions and rewards. Conducting some of this work with stakehoders is important, not ony for generating appropriate mechanisms, but for gaining ownership and consensus. Instructions Transfer the information from the deivery map into this framework. Popuate the rest of the framework in consutation with stakehoders. Stakehoders Performance current or proposed issues Recommendations evers eg. sanctions and rewards contracts, performance management etc. Name Leve Financia Operationa Reputationa Organisation 1 Organisation Team Individua Organisation 2 Organisation Team Individua Target popuation Individua The use of sanctions and rewards in the pubic sector 49

52 Annex four Performance evers Performance evers Each performance ever we have identified is discussed beow, incuding expanations of some of the impicit sanctioning or rewarding effects associated with them. Legisation. As we as estabishing forma sanctions and rewards, egisation may aso generate informa incentives. For exampe, The Environment Act 1995 requires Loca Authorities to review and assess the air quaity in their area. There are no forma sanctions with respect to high poution eves. If eves are judged as being too high, the area wi be designated an Air Quaity Management Area (AQMA). Furthermore, an action pan wi need to be set up by the Loca Authorities to reduce poution. Loca Authorities may have the impicit incentive to avoid becoming an AQMA, as this woud stigmatize the area, which in turn might ead to negative effects, such as a reduction in tourism. Measures impemented to prevent becoming an AQMA may hep the Department for Transport and the Department for Environment, Food and Rura Affairs achieve their common objective to reduce greenhouse gas emissions and improving air quaity (HM Treasury, 2007a). Taxation. Whie the primary purpose of taxation is to raise government revenue, it is often used as an impicit motivationa instrument to support the achievement of a variety of objectives. For exampe, impementing the right tax scheme may hep achieve the Department of Heath s objective to reduce adut smoking, and increases in fue duty may hep the Department for Transport and the Department for Environment, Food and Rura Affairs meet their air quaity and cimate change targets (HM Treasury, 2007a). Targets. Setting targets may create an impicit motivation for Agents to perform better. This may be because they set a cear expectation, assign accountabiity, create a fear of osing credibiity, and generate externa pressure for improved performance. For centra government, performance against Pubic Service Agreement and Vaue for Money targets is among the factors infuencing the funding received by Departments, even though the reationship between performance and funding is not expicity quantified. Reguations and standards. Reguation aims to safeguard and/or raise the quaity of service provision, and to promote equity by ensuring everyone has access to a minimum standard of provision regardess of their circumstances. For exampe, the Department of Heath s Nationa Service Framework which specifies minimum standards for heath and socia services for chidren, young peope and pregnant women. For private sector companies in reguated sectors, 50 The use of sanctions and rewards in the pubic sector

53 Performance evers annex four reguation can set benchmarks. For exampe, in its price review, Ofgem sets a price cap the path of the maximum price a company may charge for its services during the price review period. By aowing energy companies to keep the profits earned by reducing costs beow the eves assumed by the reguator, they have an impicit motivation to become more efficient. In the next price review, Ofgem wi take cost reductions from the previous period into account and potentiay be abe to set a ower maximum price path. Funding. Receipt of government funding may be ony indirecty dependent on meeting certain criteria, which are themseves not aways quantifiabe. For exampe, some of the funding for Loca Authorities is reated to the Loca Transport Pan and the counci s performance in terms of air quaity. Councis therefore have an impicit incentive to improve the quaity of their air, but without an expicit ink to funding. If a counci performs we in terms of air quaity, it heps with the achievement of both the Department for Environment, Food and Rura Affairs objective to improve air quaity, as we as the Department for Transport s objective to provide an efficient and reiabe interregiona transport system (HM Treasury, 2007a). Pubishing performance data. Even without associated sanctions or rewards, pubished data can be used to drive performance. In some cases there is an impicit motivation among empoyees and organisations to try to outperform each other. For exampe, poice forces often dispay weeky or monthy performance data within poice stations (HM Treasury, 2004b). The dispay of this data provides incentives for poice officers to increase their performance, either to compare we in reation to coeagues, or for the district to compare we in reation to other districts. Education. Exampes of which incude: (i) Active campaigning, and creating an atmosphere of strong socia disapprova for offenders of a certain norm, may set informa sanctions for peope to disobey this norm. For exampe, Siege (2006) argues that the UK campaign to shame drunk drivers created a mora cimate with strong informa sanctions. This supported achieving the Department for Transport s objective to reduce the number of peope kied or seriousy injured in road accidents; bearing in mind that about one sixth of a road casuaties invove drivers which are over the ega acoho imit (HM Treasury, 2004c). (ii) Better nutritiona information may incentivise peope to improve the quaity of their diet. This may aso set the incentive for the food industry to increase the quaity of its products which in turn woud hep the Department for Environment, Food and Rura Affairs to achieve its objective to promote a safe food suppy chain which meets consumers requirements (HM Treasury, 2004c). (iii) Ofcom requires fixed-ine teecoms operators to submit periodic reports on key performance indicators. These are made avaiabe to the pubic on the TopComm website. This aows consumers to judge the fixed ine operators based on these performance indicators and switch to the one which they fee best meets their requirements. This gives companies the incentive to improve their performance against these indicators in order to attract new, or at east not ose existing, customers. Competition and choice. Since the 1980s there have been concerted attempts to introduce more consumer choice in some pubic services. For exampe, parents have more choice over which schoos they can send their chidren to, and GPs have been given more power to seect secondary heath care providers on their patients behaf. As ong as users or patients can make informed choices, and funding foows those decisions, heathcare providers face incentives to provide high quaity, cost-effective services. However, there are probems with attempts to introduce competitive pressure to pubic services. In the case of schoos, the costs to the consumer of switching are significanty higher than in the The use of sanctions and rewards in the pubic sector 51

54 Annex four Performance evers case of, for exampe, utiity providers. Chidren from famiies that are ess we-informed, or ess mobie, are at risk of becoming stuck in ess we-performing schoos, furthering the inequity. It aso gives schoos the incentive to pick the best pupis. Bradey and Tayor (2007) found that reforms in the schoo system have increased the efficiency of secondary schoos (measured in terms of exam performance) but at the expense of increased socia segregation. Performance and staff management. Reguar performance reviews can act as strong incentives for deivery chain bodies to maintain and improve service eves. For exampe, at the centra government eve, a Departments are subject to Capabiity Reviews, which require the pubication of an action pan by the Department. This incentivises Departments to improve performance, even though the consequences of faiing to achieve the stated improvements are not made expicit. On staff remuneration, in most cases in the pubic sector, individua performance is assessed against a range of non-quantifiabe objectives. But even where the forma performance-reated component of overa remuneration is sma as it typicay is in the pubic sector performance can be incentivised by the impicit promise of onger-term career progression. Contracts. In many cases, contracts wi expicity set out the financia and other impications of performance faing beow, or exceeding, target eves. However, contracting bodies can aso use a range of impicit sanctions and rewards, such as the prospect of a contract extension subject to satisfactory performance. There are aso the reputationa consequences to the Agent of having their contract terminated, or not extended. Partnerships and Memoranda of Understanding. At the centra government eve, joint objectives can be supported by deivery pans and monitoring systems. In other cases, joint working may be driven by shorter term imperatives, such as the need to respond to a crisis, ike fooding. At the oca eve, partnerships between Loca Authorities and key stakehoders have become increasingy common, either on a biatera basis (such as between a Loca Authority and a further education coege) or coectivey, through umbrea arrangements, such as Loca Strategic Partnerships. Another exampe might be the shared objectives of a prison and a drugs rehabiitation charity. These reationships are not aways contractuay based; NAO (2006) notes that bus use in Brighton has grown strongy despite the absence of a direct contract between the bus operators and the Loca Authority. 52 The use of sanctions and rewards in the pubic sector

55 Performance evers annex four The use of sanctions and rewards in the pubic sector 53

56 annex five Bibiography Bibiography Armstrong, M. & Muris, H. (1998). Reward Management, 4th ed. London: Kogan Page. Armstrong, M. (1999). Empoyee Reward, 2nd ed. Institute of Personne and Deveopment. Armstrong, M. & Brown, D. (2001). New Dimension in Pay Management. CIPD. Atkinson, A., Burgess, S., Croxson, B., Gregg, P., Propper, C., Sater, H., Wison, D. (2004). Evauating the Impact of Performance-reated Pay for Teachers in Engand. CMPO Working Paper Series No. 04/113. Bradey, S., & Tayor, J. (2007). Diversity, choice and the quasi market: An empirica anaysis of secondary education poicy in Engand. Department of Economics, Lancaster University. Burgess, S., Metcafe, P. (1999) The use of incentive schemes in the Pubic and Private sectors: Evidence from British Estabishments. CMPO Working Paper 99(015). Burgess, A., Propper, C. & Wison, D. (2002). Does performance monitoring work? A Review of the evidence from the UK Pubic sector, excuding heathcare. CMPO Working Paper No. 02/49, CMPO, University of Bristo. Burgess, S., & Ratto, M. (2003). The roe of incentives in the Pubic Sector: Issues and Evidence. CMPO Working Paper 03/071, CMPO, University of Bristo. Burgess, S., Propper, C., Ratto, M., & Tominey, E. (2004). Evauation of the Introduction of the Makinson Incentive Scheme in Jobcentre Pus. CMPO, University of Bristo. Burgess, S., Propper, C., Ratto, M., Schoder, S. & Tominey, E. (2005). Evauation of the Introduction of the Makinson Incentive Scheme in HM Customs and Excise. CMPO, University of Bristo. Cabinet Office. (2006). The UK Government s Approach to Pubic Service Reform. Communities and Loca Government. (2006). Strong and Prosperous Communities The Loca Government White Paper. Dearden, L., Emmerson, C., Frayne, C., Meghir, C. (2004). Can Education Subsidies stop Schoo Drop-outs? An evauation of Education Maintenance Aowances in Engand. Institute for Fisca Studies. Government Accountabiity Office (GAO). (2006). Grants Management: Enhancing Performance Accountabiity Provisions Coud Lead to Better Resuts. GAO Gneezy, U. (2003). The W effect of incentives. The University of Chicago Graduate Schoo of Business. Grout, P.A., & Stevens, M. (2003). Financing and managing pubic services: an assessment. CMPO Working Paper No. 03/076, CMPO, University of Bristo. 54 The use of sanctions and rewards in the pubic sector

57 Bibiography Annex five HM Treasury. (2004a). Devoving Decision Making: 1 Deivering better pubic services: refining targets and performance management. HM Treasury. (2004b). Devoving Decision Making: a consutation on regiona funding aocations. HM Treasury. (2004c). SR 2004: pubic service agreements 2005/2008. HM Treasury. (2007a) Pre-Budget Report and Comprehensive Spending Review Meeting the aspirations of the British peope. HM Treasury. (2007b). Review of sub-nationa economic deveopment and regeneration. Pubished in coaboration with the Department for Business Enterprise and Reguatory Reform, and Communities and Loca Government. Heckman, J., Heinrich, C., & Smith, J. (2002). The Performance of Performance Standards. Journa of Human Resources, 37(4), Hibbard, J., Stockard, J., Tuser, M. (2003). Does Pubicizing Hospita Performance Stimuate Quaity Improvement Efforts?. Heath Affairs, Voume 22, Number 2. Kahn, C., Siva, E., & Ziiak, J. (2001). Performance-Based Wages in Tax Coection: The Braziian Tax Coection Reform and Its Effects. Economic Journa, 468, Keman, S., Friedman, J. N. (2007). Performance Improvement and Performance Dysfunction; An Empirica Examination of Impacts of the Emergency Room Wait-Time Target in the Engish NHS. Harvard University JFK Schoo of Government Facuty Research Working Paper RWP Le Grand, J. (1995). Knights, Knaves or Pawns? Human Behaviour and Socia Poicy. Journa of Socia Poicy, 26(2), Levitt, S.D., & Dubner, S.J. (2005). Freakonomics: A Rogue Economist Expores the Hidden Side of Everything. New York: HarperCoins Pubishers. The Lewin Group & ECONorthwest (2003). Study of the Impementation of the Performance-Based Incentive System. Macrory, R. (2006). Reguatory Justice: Making Sanctions Effective. London: Better Reguation Executive. Makinson, J. (2000). Incentives for change: Rewarding performance in nationa government networks. Pubic Services Productivity Pane, HMT. Marsden, D. & French, S. (2002). Performance Pay in the United Kingdom , in Brown, M., Heywood, J. S. (Eds.). (2002). Paying for Performance: An Internationa Comparison. New York, London: ME Sharpe. Migrom, P., & Roberts, J. (1990). The economics of modern manufacturing: technoogy, strategy and organization. American Economic Review, 80(3), Nationa Audit Office (NAO). (2001a). Measuring the Performance of Government Departments. HC 301, Nationa Audit Office (NAO). (2001b). Inappropriate adjustments to NHS waiting ists. HC 452, Nationa Audit Office (NAO). ( ). Pubic Service Agreement: Vaidation Compendium Reports 1-4. HC 476, HC 985, HC 127 (I & II), HC 22 (I & II) The use of sanctions and rewards in the pubic sector 55

58 annex five Bibiography Nationa Audit Office (NAO). (2006). Deivering Efficienty: Strengthening the inks in pubic service deivery chains. Pubished in coaboration with the Audit Commission. HC 940, Nationa Audit Office (NAO). (2007). Targeting Inequaities. Nationa Audit Office (NAO). (2008). NHS Pay Modernisation: New Contracts for Genera Practice Services in Engand. HC 307, Prendergast, C. (2001). Seection and oversight in the pubic sector, with the Los Angees Poice department as an exampe. Nationa Bureau of Economic Research Working Paper 37. Prentice, G., Burgess, S., & Propper, C. (2007). Performance-pay in the pubic sector: A Review of the issues and evidence. Office of Manpower Economics. Propper, C. & Wison, D. (2003). The Use and Usefuness of Performance Measures in the Pubic Sector. CMPO Working Paper No. 03/073, CMPO, University of Bristo. Propper, C., Sutton, M., Whitna, C., & Windmeijer, F. (2007). Did targets and terror Reduce Waiting Times in Engand for Hospita Care?. CMPO Working Paper 07/179, CMPO, University of Bristo. Seabright, P. (2004). Continuous Preferences Can Cause Discontinuous Choices: An Appication to the Impact of Incentives on Atruism. Mimeo, IDEI, Tououse University. Siege, L.B. (2006). Criminoogy: Theories, Patterns & Typoogies 9th ed. Wadsworth Pubishing. 56 The use of sanctions and rewards in the pubic sector

59 This report can be found on the Nationa Audit Office web site at For further information about the Nationa Audit Office pease contact: Nationa Audit Office Press Office Buckingham Paace Road Victoria London SW1W 9SP Te: Emai:

60 Design and Production by NAO Marketing & Communications Team DG Ref: Printed by SLS Print Nationa Audit Office 2008