Trying It Out. The Role of Pilots in Policy-Making. Report of a Review of Government Pilots. Government Chief Social Researcher s Office

Transcription

1 Trying It Out The Role of Pilots in Policy-Making Report of a Review of Government Pilots Government Chief Social Researcher s Office

2 This report was produced on behalf of the Cabinet Office by Professor Roger Jowell, City University. The views expressed in this report are the author s own and do not necessarily reflect those of the Cabinet Office. December 2003 Government Chief Social Researcher s Office ISBN Crown Copyright 2003

3 FOREWORD & ACKNOWLEDGEMENTS by Roger Jowell, Chair of the Review Panel This report has relied on a number of important contributions. Although the primary responsibility for its final shape and content is mine, I am aware of how much I have depended on others for important information, data and ideas. Before the Review Panel had even met, the Strategy Unit invited a distinguished group of academics, practitioners and civil servants to a seminar whose aim was to guide our agenda. Looking back, that seminar not only raised many of the key issues we subsequently attempted to tackle in our deliberations, but as importantly helped us to avoid tempting cul de sacs. We are grateful for the time and wisdom of those who attended (see Annex 1). As soon as the Panel (p.4 and p.40) began its deliberations, it confirmed my impression of how much more many of its members knew about the detail of the subject than I did. Some had written influential articles and books about policy pilots. Others had personally overseen the implementation of one or more pilots within departments. All had a clear picture of the advantages, disadvantages and potential pitfalls of piloting. I am indebted to them for the cogent ideas and arguments they brought to the table and for their later criticisms and proposed edits of early drafts. This report comes from the full Panel. We drew on a number of sources, including a literature review, a postal survey of policy makers and researchers in nine departments, and face-to-face interviews with a selection of these respondents, as well as a handful of Ministers. We concentrated on people who had themselves had personal experience of one or more policy pilots. As expected, these questionnaires and interviews produced intriguingly different perspectives of the process itself and its inevitable tensions. On the basis of these data and the literature search, we then assembled a series of illustrative case studies that appear throughout the body of this report. The smooth implementation of all this work was entrusted to staff within the Prime Minister s Strategy Unit. The project was initiated by Sue Duncan (whom we subsequently co-opted onto the Panel itself) and Phil Davies. Initial support was provided briefly by Stephen Morris and for a longer period by Rebecca Stanley before both moved on to other roles, but not before making valuable contributions particularly to the shape and structure of the work. This left Annette King to see most of the project through with great energy and skill, acting both as the Panel s secretary and the Chair s ankle-biter until our work was well and truly done. She played a vital FOREWORD & ACKNOWLEDGEMENTS

4 role in bringing this report to fruition. A special tribute is also due to Phil Davies under whose watchful, observant and knowledgeable eye Rebecca, Annette and the Panel itself all worked and learned. Tess Ridge of the University of Bath joined the team temporarily to undertake the excellent literature review, and Lucy Woodward also a temporary member of the team skillfully assembled the case studies. Their work greatly eased ours. Finally, although they must as usual remain anonymous in a report of this kind, we are deeply indebted to the civil servants and Ministers who patiently and frankly answered all our questions, providing us with unique insights into the provenance, conduct and aftermath of policy piloting in a range of different circumstances. Their thoughtful insights helped not only to inform the report as a whole but also to influence our recommendations (p.5). RJ (December 2003) Trying It Out The Role of Pilots in Policy-Making

5 CONTENTS Foreword and acknowledgments 1. Introduction 3 2. Recommendations 5 3. The case for piloting Types of pilot Properties of pilots What s in a name? Pilots as insurance policies Pilots to test variance Examples of impact and process pilots Timing of pilots Pilot methodologies Alternative approaches Experimental methods randomised controlled trials (RCTs) Ethical considerations and RCTs Quasi-experimental methods The US perspective Widespread use of RCTs Some exceptions and reservations Contrasts between the US and Britain How much influence do pilots have? Experience in the UK survey responses Range and spread of pilots Political v. research imperatives Timetable imperatives Transparency of results Who decides? Design considerations Resource considerations Technical considerations Promoting innovation 32 Contents 1

6 7. Conclusion References 36 Annex 1: Workshop participants 40 Annex 2: Methods Project organisation Approach 42 Annex 3: The Interview guide 44 Trying It Out The Role of Pilots in Policy-Making 2

7 1. INTRODUCTION An important innovation in recent years has been the phased introduction of major government policies or programmes, allowing them to be tested, evaluated and adjusted where necessary, before being rolled out nationally. This practice has been widespread in the USA for much longer, partly because its federal structure enables individual states to mount their own fairly large-scale experimental pilots to test the likely impact of a proposed new policy or delivery mechanism or both (Greenberg and Shroder, 1997). The impact of such pilots in the US has been mixed sometimes helping to prove certain policies, sometimes leading to adjustments of either policy or process, and sometimes to their abandonment. The sharp growth in the number and scale of British pilots since 1997 (Walker, 2001; Sanderson, 2002) led to a call in the wideranging report on modernising government, Adding It Up (Performance and Innovation Unit, 2000) for their methods and fitness for purpose to be evaluated. The report recommended 'more and better use of pilots to test the impacts of policies before national roll-out'. To help achieve this aim, it also recommended the creation of a panel of enquiry to oversee an exchange of experiences between departments across UK administrations and to consider the future role of pilots. The Government Chief Social Researcher s Office (GCSRO) in the Strategy Unit was given responsibility for setting up this panel (see membership below), which began its work in September It met three times and initiated the following set of activities, the output from which forms the basis of this report: a workshop of experts in the field of social policy evaluation and piloting to help develop and shape the framework for, and scope of, the review; a literature review charting the experience of successful (and unsuccessful) policy pilots both in the UK and abroad, and summarising key academic and professional debates about their role; a self-completion questionnaire sent to the heads of research in key government departments across UK administrations to help map the scale and types of pilots that had been carried out in the UK over the last five years and their perceived impact; face-to-face interviews with senior civil servants in both research and policy roles to explore their experience of piloting of different kinds; face-to-face interviews with selected Ministers to discover their own perspective on recent pilots within their ministries and 1. Introduction 3

8 case studies from government departments across UK administrations to illustrate a range of approaches to piloting. Review Panel* Professor Roger Jowell, Chair Professor Waqar Ahmad Sue Duncan Professor John Fox Professor Edward Page Review Team* Phil Davies Annette King Rebecca Stanley Tess Ridge Lucy Woodward Michael Richardson Judy Sebba Ann Taggart Professor Robert Walker Professor Paul Wiles Trying It Out The Role of Pilots in Policy-Making *See Annex 2 for affiliations and further details of the work undertaken. 4

9 2. RECOMMENDATIONS 1 The role of pilots 1. The full-scale introduction of new policies and delivery mechanisms should, wherever possible, be preceded by closely monitored pilots. Phased introductions help not only to inform implementation but also to identify and prevent unintended consequences. A pilot is an important first stage of regular, longer-term policy monitoring and evaluation. (3.1; 3.2; 3.4; 3.7; 6.5; 6.6) 2. Although pilots or policy trials may be costly in time and resources and may carry political risks, they should be balanced against greater risk of embedding preventable flaws into a new policy. Initial policy submissions to Ministers should explicitly consider such factors and contain a section on possible piloting strategies. (3.4; 3.5; 6.7) 3. Advantage should be taken of the small scale and explicitly experimental nature of pilots to encourage innovations in policy that might otherwise be too risky or costly to embark on. (3.4; 3.6; 6.9) 4. Pilots should vary in their nature and scope according to a range of factors not all of which are obstacles such as tight timetables or low budgets. Also important in shaping a piloting strategy should be the extent of Numbers in italics denote cross-references to relevant sections in the main report. accumulated knowledge already available about that policy area. The scale and complexity of any experimental treatment should be proportionate to its likely utility. (3.7; 6.1; 6.2; 6.3; 6.6) Pre-conditions of pilots 5. A pilot should be undertaken in a spirit of experimentation. So, if it is clear at the outset that a new policy and its delivery mechanisms are effectively already cast in stone, a pilot is redundant and ought not to be undertaken. (3.2; 6.7) 6. Once embarked upon, a pilot must be allowed to run its course. Notwithstanding the familiar pressures of government timetables, the full benefits of a policy pilot will not be realised if the policy is rolled out before the results of the pilot have been absorbed and acted upon. Early results may give a misleading picture. (3.2; 3.4; 3.7; 6.2; 6.3; 6.8) 7. Many policies take time to bed in; others are intended to achieve only modest changes in outcomes. The timetable and scale of a pilot must take account of such factors so as to avoid producing a false impression of policy failure. (3.7; 6.3; 6.8) 2. RECOMMENDATIONS 5

10 Trying It Out The Role of Pilots in Policy-Making 8. As with all policy development, pilots should be preceded by the systematic gathering of evidence from the UK and abroad. (3.7; 6.6) 9. The precise purpose(s) of a policy trial whether it is to measure a policy s likely impact or to test its delivery mechanisms, or both must be made explicit in advance so that its methods and timetable are framed accordingly. (3.1; 3.2; 6.1; 6.6) Key properties of pilots 10. Independence is critical. Pilots must be free from real or perceived pressure to deliver good news and be designed to bring out rather than conceal a policy s imperfections. To this end, the Ministers and civil servants most closely involved with the policy should consider distancing themselves from decisions about pilot methods and the dissemination of their findings. (3.7; 6.4; 6.5) 11. Methods matter. A poorly conceived or poorly specified pilot may be worse than no pilot at all. To ensure that the methodology of a pilot is as bullet-proof as possible, expert internal and external advice should be drawn on early, and appropriate resources made available. (4.1; 4.2; 4.4; 6.6; 6.7; 6.8) 12. Nomenclature matters too. The terms pilot and policy trial should be reserved for rigorous early evaluations of a policy or some of its elements rather than for other forms of research into a policy s early performance. (3.1; 3.2; 3.3) Numbers in italics denote cross-references to relevant sections in the main report. 13. Tags such as trailblazer or pathfinder are best avoided for genuine pilots or policy trials. By creating unrealistic expectations, they tend to make neutral evaluation more difficult. (3.3) 14. It must be recognised that the policy process is not always suited to rigorous and necessarily lengthy pilots in advance of a policy roll-out. Time and resources are limited and Ministers are often impatient to deliver. So provision for interim findings always accompanied by appropriate health warnings must be anticipated. (3.7; 6.2; 6.3; 6.6; 6.7) 15. To avoid systematic errors in the conduct of pilots, their budgets and timetables should allow for adequate training of the staff who are to administer processes such as allocating participants to treatment and control groups. Policy and research staff training should also include modules on piloting and evaluation. (6.7) Methods and practices of piloting 16. There is no single best method of piloting a policy. Multiple methods of measurement and assessment including experimental, quasi-experimental and qualitative techniques should all be considered to get a complete picture. (4.1; 4.2; 4.4) 17. For policies designed to achieve change in individual behaviour or outcomes, randomised controlled trials of individuals offer the most conclusive test of their likely impact. Long underused in the UK, they should more often be considered as vehicles for rigorous trials. (4.2; 5.1; 5.2; 5.3) 6

11 18. For policies designed to achieve change at an area, unit or service level (such as in schools, hospitals or job centres), randomised area- or servicebased trials offer the most conclusive test of impact and should more often be used in preference to non-random (matched) trials. (3.6; 4.2; 4.4) 19. However, since random allocation is sometimes impracticable and unsuited to addressing certain questions (such as why a particular outcome may have occurred), a battery of other techniques should also be considered, either on their own or in tandem. (4.4) 20. Rigour is by no means confined to the quantitative testing of new policy initiatives. Well-founded qualitative research among both beneficiaries and service providers should also feature in a comprehensive pilot. (4.1; 4.4) 21. The ethical demands of pilots cannot all be met via informed consent from participants. Inequities between beneficiaries and non-beneficiaries and the risk of negative consequences for some participants both need attention. Such problems should, however, be addressed and mitigated rather than treated as insuperable obstacles to rigorous experimentation. (4.3; 6.3) 23. Pilots should be regarded less as ad hoc evaluations than as early stages in a continuing process of accumulating policy-relevant evidence. (6.3; 6.6) 24. Appropriate mechanisms should always be in place to adapt (or abandon) a policy or its delivery mechanisms in the light of a pilot s findings. (3.2; 3.4; 3.7) 25. To ensure the effective exploitation of policy-relevant evidence, departmental dissemination strategies should ensure that both the results and methods of pilots are made freely available within and outside government. (6.4; 6.7) 26. Post-pilot reviews should be routinely undertaken and published as a means of sharing experience and developing methods. (6.3; 6.7) 27. An accessible central electronic repository of pilot reports should be set up to facilitate easy reference to past successes and failures. (4.4; 6.7) Using pilot results 22. A pilot that reveals a policy to be flawed or ineffective should be viewed as a success rather than a failure, having potentially helped to avert a potentially larger political and/or financial embarrassment. (3.4) 2. RECOMMENDATIONS 7

12 3. THE CASE FOR PILOTING Trying It Out The Role of Pilots in Policy-Making 3.1 Types of pilot The Adding It Up report (Performance and Innovation Unit, 2000) referred to two ways in which piloting is undertaken within government: Impact pilots are tests of the likely effects of new policies, measuring or assessing their early outcomes. They enable 'evidence of the effects of a policy change to be tested against a genuine counterfactual, such as is provided by the use of control groups in a medical trial'. Process pilots on the other hand are designed to explore the practicalities of implementing a policy in a particular way or by a particular route, assessing what methods of delivery work best or are most cost-effective. The boundary between these two broad types of pilot is often blurred and many pilots seek to achieve both aims. In addition to investigating whether a new policy intervention will actually work (i.e. an impact pilot), some pilots try to acquire evidence about who it will and will not work for, at what financial and social cost, and whether it might work more effectively via a different route (i.e. a process pilot). Impact and process pilots are also sometimes used to help improve an existing policy or its methods of implementation, or to develop a new policy from a preliminary idea. 3.2 Properties of pilots Many new policy initiatives throughout government are now being introduced in distinct phases, in principle, to enable their effectiveness to be tested in advance of their full-scale implementation. The most common form of phased implementation is initially to introduce a new policy within only a limited number of test areas (ideally, but not always, randomly selected ones). On occasion, a new policy initiative may instead be randomly allocated to a small group of individuals in advance of being rolled out nationally. Either way, the relatively small scale and the experimental nature of such pilots can combine to produce a rigorous early assessment of a policy s likely effectiveness and, ideally, how it can be improved before it is cast in stone. Based broadly on well-established methods of medical experimentation (Cochrane, 1972), the impact of a new policy is measured by comparing the test population against controls that have been selected in precisely the same way but have not (yet) benefited from the treatment (Campbell and Russo, 1999). A policy pilot should be seen above all as a test run the results of which will help to influence the shape and delivery of the final policy. It follows that a policy pilot must be allowed to run its course and produce its findings before the policy is rolled out. Too often, this has not been the case. Interim results will provide useful feedback on early impact and may highlight delivery issues 8

13 Case Study 1 Employment Retention and Advancement (ERA) Scheme: Design Phase Cabinet Office Aim: A trial of the effectiveness of new services to improve job retention and advancement prospects for low-wage workers. Background: Some groups of low-waged workers in the UK face uncertain and unstable employment prospects. They tend to work in sectors paying the lowest wages and remain on the margins of the labour market. They are likely to face recurring periods of unemployment, or under-employment, and have poor prospects for improving their earnings. Methods: Advancement and support advisers will provide new services and financial incentives to help low-waged workers remain in employment for longer (retention) and have a better chance of increasing their earnings and other working conditions over time (advancement). The aim is to measure whether workers receiving ERA services retain work and advance in employment to a greater extent than if they had not received ERA services, as well as to assess the costs and benefits of the policy. Phase 1 involved the design of the policy and an evaluation strategy by a team with expertise in policy, evaluation and implementation. The team worked as an independent group with consultants and stakeholders involved throughout. Phase 2 will be to implement the policy led by the Department for Work and Pensions. In Phase 2, the project will run in six demonstrator sites and offer new services to those eligible for New Deal; those volunteering for New Deal for Lone Parents and Lone Parents on Working Tax Credit, working part-time. The evaluation comprises an impact assessment using random assignment methods; a process study and a cost benefit analysis. The key objectives are: to determine whether, and to what extent, the new measures improve employment stability and advancement; to identify the costs and benefits of the policy to participants, employers, the exchequer and society as a whole; and to identify lessons for the implementation of the policy. Lessons learned: The key lessons from the design phase highlighted the importance of effective project organisation and working structures, especially the need for a multi-disciplinary team comprising policy-makers, implementation experts and analysts. Lessons learned from designing and running demonstration pilots over 25 years in the US and Canada were useful precedents for the ERA project. Contact details/further information: Dr Phil Davies (Cabinet Office), Tel: , Kellard, K., Adelman, L., Cebulla, A, and Heaver, C. (2002), From Job Seekers to Job Keepers: Job Retention, Advancement and the Role of In-Work Support Programmes, DWP Research Report 170, London: Department for Work and Pensions. Morris, St., Greenberg, D., Riccio, J., Mittra, B., Green, H., Lissenburg, S. and Blundell (2003), The United Kingdom Employment Retention and Advancement Demonstration Design Phase: An Evaluation Design, GCSRO Occasional Papers Series No.1, London: Government Chief Social Researcher's Office, Cabinet Office. which need attention. But they may give a misleading picture of long-term policy outcomes. Early roll-out before the full policy impact is clear reduces the value that can be derived from the piloting process and carries the risk of policy failures. The British legislative process is, in practice, not very conducive to genuine piloting. By the time a policy has reached the statute books, its content (and often its methods of delivery too) have run the gauntlet of parliamentary debate, media examination, pressure from lobbies and scrutiny by committees. Emerging from this 3. THE CASE FOR PILOTING 9

14 Trying It Out The Role of Pilots in Policy-Making process, the final version of a policy may well incorporate numerous carefully worked compromises which are by then far too complex to be re-opened. There are of course notable exceptions, such as the present Employment Retention and Advancement (ERA) Scheme (Case study 1 p.9) (Morris et al., 2003), which is explicitly designed to influence the existence and shape of legislation. Developed by a team in GCSRO, it is being carried out by the Department for Work and Pensions (DWP). The work is being undertaken in advance of a fixed policy commitment. In other recent cases, however, it has been clear from the outset that a pilot would have no real chance of influencing policy or delivery in time for it to matter. The policy itself and its delivery mechanisms were already so firmly in place that a pilot was effectively redundant. The same is often true of policy initiatives that are based directly or indirectly on prior manifesto commitments. When such policies eventually reach the statute books, they tend not only to have long been heralded as important new departures, but also often to carry the weight of ministerial or even governmental reputation. In these circumstances, the imperative is understandably to achieve their smooth and successful implementation, unencumbered by unwelcome news of the sort that suggests that the policy may incorporate flaws after all. While such high-profile policies would in several respects benefit particularly from cautious piloting followed by judicious finetuning, it is equally clear why any Minister wants to avoid the risk that a cherished policy will fall at the last hurdle as a result of research that he or she has commissioned. 3.3 What s in a name? In our discussions with civil servants and Ministers, we discovered considerable confusion about the distinction between the different policy-testing mechanisms commonly employed within government. Some had been referred to as pilots when they were patently not pilots, because the policy and its delivery mechanisms were already well and truly fixed from the outset. True, these early evaluations of a policy s impact or process would doubtless come into their own one day, but what was absent at the time was a spirit of experimentation, unburdened by promises of success. By the same token, other forms of phased implementation of policies had unaccountably not been referred to as pilots, even though they had, in fact, been designed as neutral trials of policy or process with at least some chance of influencing the final product. They attracted tags such as pathfinders, trailblazers, pioneers, prototypes or benchmarks names which implied, wrongly, that they were innovative exemplars rather than rigorous policy trials. Not only do departments across UK administrations differ in their use of labels, but so too do divisions of the same department or administration, compounding the confusion. Fanciful terms for early evaluations of policy have multiplied to the extent that one of the Ministers we interviewed reported having been given the option to choose the tag that he or she liked best for a pilot from a range of competing but equally inappropriate options. If there was one almost universal demand of this Review, it was to help clarify the present fog in relation to nomenclature. 10

15 Our advice is simple. We favour describing early evaluations in relatively mundane but accurate terms. Dressing them up as described above is a counterproductive distraction. The term pilot should ideally be reserved for 'rigorous early evaluations of a policy (or some of its elements) before that policy has been rolled out nationally and while is still open to adjustment in the light of the evidence compiled'. Also, in the interests of transparency, a pilot should be classified in advance as to whether its purpose is to assess impact or test process or both. To avoid creating false expectations, other forms of research into a policy s early performance should not be described as piloting and should in any case for much the same reason shun fashionable tags such as pathfinder, trailblazer and the like. 3.4 Pilots as insurance policies In an ideal world, all pilots would probably be policy development pilots and would take place in an orderly way well before a particular policy decision was formulated (Mandell et al., in press). As noted, however, our political system makes this difficult. That is not to suggest that policy-making is not well founded; other sources of evidence of policy outcomes may be available. For example, some policies have been informed by evidence from analogous policy interventions in Britain and abroad (Lipsey and Wilson, 1993). Others have been informed by prior research on the same general topic. Still, even when the case for a particular policy intervention has been comprehensively made, and even when there is an explicit manifesto or other commitment to introduce it, we would still recommend piloting in advance of its full-scale implementation. If nothing else, it helps to identify and mitigate unintended consequences, such as negative impacts on certain subgroups. More generally, a pilot helps to eliminate fault lines in a policy in rather more propitious circumstances than after it has been comprehensively rolled out with an accompanying fanfare. Above all, pilots serve the cause of evidenceinformed policy and cost-effective delivery mechanisms. They help to protect Ministers, governments and taxpayers from potentially expensive failures. Many policy interventions are introduced in the face of robust parliamentary and sometimes public opposition, and few will achieve with equal success all of their often wide-ranging and ambitious aims. A properly conducted pilot acts as an invaluable defence mechanism, not only against the risk of a well-intended intervention going spectacularly awry, but also against its going slightly wrong in a patently preventable way. By conducting systematic experimentation in advance of the full-scale implementation of their policy interventions, governments are simply falling into line with practice in almost all other fields. Prior testing makes innovation less risky and therefore more likely (6.9), though it must be admitted that, on occasion, even testing an unpopular policy may attract flak. A recent example of a controversial policy innovation which might not have been politically possible in the absence of a successful experiment is the Office of the Deputy Prime Minister s (ODPM) Pilot Seller s Information Pack (Case study 2, p.12) (ODPM, 2002). The presence of solid evidence in advance of implementation helped not only to reassure the proponents of the policy, but also to placate some of its opponents (Greenberg et al., 2000; Sanderson, 2002; Walker, 2001; Mandell et al., in press). 3. THE CASE FOR PILOTING 11

16 In common with certain Royal Commissions and other long-term enquiries or research projects, pilots may sometimes be used simply as a means of delaying a policy decision. Usually referred to as the long grass mechanism, controversial action may be deferred in the expectation either that sufficient political will or resources will materialise in due course, or that the problem will just eventually go away. Case Study 2 Pilot Seller s Information Pack Office of the Deputy Prime Minister Aim: Pilot to assess the practicalities of assembling a seller s pack (now referred to as a home information pack) and the difference it made to the process of buying and selling a home, and to inform decisions on a national scheme. Background: Home buying and selling in England and Wales is inefficient, wasteful and among the slowest in Europe. An important factor in this is that important information about the property only becomes available at a later stage in the transaction, which can cause delay or failure of the sale. The home information pack provides important information before the property is put on the market. Methods: The pilot aimed to test if greater certainty in the home buying and selling process could reduce delays and the number of abortive transactions. The scheme involved 159 houses and flats offered for sale by private owners and the sale of 30 new home plots being sold by Beazer Homes. The pack contained searches, evidence of title, a property information form containing the seller's replies to standard pre-contract questions, a summary of the contract, copies of guarantees and warranties and a report on the condition and energy performance of the property. The pack was distributed through 31 estate agent offices. The views of those involved in the home buying and selling process were collected through surveys: regular telephone calls to sellers; a survey of conveyancers on each transaction coming under offer; and in-depth interviews with buyers and sellers. All the key stakeholders were involved in helping to formulate and interpret the results. The pilot's results were compared with the earlier Housing Market Transactions Study. Findings: The pilot provided clear evidence that the scheme produced real benefits to the consumer, including greater certainty, the exposure of transaction-threatening problems earlier in the process and thus less likelihood of failure later on. Trying It Out The Role of Pilots in Policy-Making Lessons learned: The pilot demonstrated that the home information pack improved the home buying and selling process for the consumer and identified areas where further changes were required, for example, changes to the report on the condition of the property being sold. The early sign-up and commitment of stakeholders and consultants was crucial. Availability of results throughout the pilots allowed important refinements to the seller's pack in the course of the pilot. Contact details/further information: Denis Purshouse, Office of the Deputy Prime Minister, denis.purshouse@odpm.gsi.gov.uk Department for Transport, Local Government and the Regions (2000), Evaluation of a Pilot Seller s Information Pack: The Bristol Scheme, Summary Report, Department for Transport, Local Government and the Regions. Office of the Deputy Prime Minister (2002), Evaluation of a Pilot Seller s Information Pack: The Bristol Scheme, Final Report, London: Office of the Deputy Prime Minister. 12

17 3.5 Pilots to test variance Certain pilots are designed to test whether the impact of a new policy is likely to vary significantly between different regions, countries or even different parts of the world. For instance, the Scottish Executive piloted Drug Treatment and Testing Orders (Case study 3, p.15) even though they had been thoroughly piloted in England (Eley et al., 2002). Similarly, the Seller s Information Pack was piloted in England, despite a good deal of evidence from abroad of its likely benefits (ODPM), 2002; Case study 2, p.12). The ERA project (a policy development pilot) (Case study 1, p.9) has been set up largely because existing evidence from similar experiments in the US and Canada is considered to be inconclusive. So, in advance of any firm commitment to a particular set of interventions, its purpose is to establish both the likely impact and the best forms of delivery of certain measures such as personal advisers, tax incentives and training bonuses which have previously been used for other purposes or among different populations. The question being addressed by the trial is effectively how well (if at all) each of these methods will work in helping to retain and advance low-income workers in the labour market (Morris et al., 2003). 3.6 Examples of impact and process pilots An excellent example of an impact pilot is the Public Defenders Pilot still being carried out by the Lord Chancellor s Department (LCD) to test the effect of salaried defenders within the English criminal justice system. There are no promises to roll out the policy unless the benefits are shown to outweigh the costs. In other cases, such as the New Deal for Lone Parents (NDLP) (Case study 4, p.18) and the Education Maintenance Allowance (EMA) (Case Study 5, p.22), while the broad policy commitment had been more or less fixed in advance, serious questions remained about their likely impact (Ashworth et al., 2002; Hales et al., 2000; Hasluck, 2000; Heaver et al., 2002; Legard et al., 2001; Maguire and Maguire, 2003). The pilots were designed to measure their early effects in certain geographical areas to which the policy was confined initially. Matched comparison areas which did not receive the treatment were selected to determine the counterfactual. In both these cases, the policy was subsequently rolled out nationally. In contrast, the substantial Earnings Top-Up policy pilot (ETU) (Case study 6, p.26) did not lead to a national roll-out. Instead, after several years of piloting designed to help test and fine-tune the policy, a general election and a change of Government intervened. The result was that other measures with similar aims such as the Working Tax Credit and the National Minimum Wage were preferred (Department of Social Security, 1996; Smith and Dorsett, 2001). Nevertheless, the ETU pilot helped to improve the design of subsequent policies in this area. Meanwhile, process pilots have also been carried out in a number of government departments across administrations such as the Department for Transport, the Office of the Deputy Prime Minister (ODPM), the Department for Education and Skills (DfES), and the Welsh Assembly Government. In each case, new policies have deliberately been introduced in phases purely, or at least mainly, as a means of refining their system of delivery, thereby reducing the risk of building in preventable flaws (also see Case study 7, p.30). 3. THE CASE FOR PILOTING 13

18 Trying It Out The Role of Pilots in Policy-Making The New Deal for Communities, for instance, was initially introduced hurriedly in 29 areas and only later rolled out in a further 50-plus areas, by which time a piloting strategy had been developed. Via both local and national evaluation, it proved possible to build improvement into the process. Practical lessons from the early area-based initiatives were similarly built into later models. The Sure Start programme of the DfES is employing a similar phased-implementation approach. The sheer magnitude of this programme would, in any event, have ruled out a full national roll-out, so the opportunity is being taken to learn from successes and failures as the programme develops which has predictable risks for consistency of measurement. Nonetheless, numerous local pilots have been set up to test innovations over an extended period, enabling the initial approaches to be restructured as appropriate. A variation of this model involves the introduction of a policy to certain population groups in advance of others. There are, however, certain risks to this approach, based as it is on the assumption that what works for a particular subgroup will necessarily work in the same way for other different subgroups. 3.7 Timing of pilots Despite the increasing use of pilots in Britain, by no means all new policies are either implemented in phases or subjected to early evaluation. We asked senior government researchers and policy people how and by whom decisions were taken on whether or not to opt for phased implementation. Some reported that the decision usually followed a systematic review of existing evidence, others that it was discussed fully at brainstorming sessions, others that it was increasingly becoming a presumption that new ideas and initiatives would be introduced in a phased way and monitored throughout. None reported the existence of a set of underlying principles that helped to guide these decisions. Departments need to take powers before a pilot can be organised, thus introducing inevitable delays. Respondents did not refer to this as a major problem. Instead, most decisions about the introduction and monitoring of new policies were preceded by a discrete judgement (whether by Ministers or senior civil servants) based largely on pragmatic considerations the most salient of which was the time frame available. The roll-out of many new policies was widely acknowledged to be governed by timetables quite unable to accommodate lengthy policy trials. Indeed, in view of the scant use likely to be made of certain pilot results and the considerable pressure on departmental analytical resources, pilots were sometimes regarded as a dispensable luxury. Once a major new policy had been announced, with its accompanying fanfare, the political and practical momentum in favour of rolling it out nationally both without delay and without modification was sometimes impossible to resist. Preventable flaws were thus sometimes built into policies and had to be rectified at more expense (and sometimes with more embarrassment) only much later. The timetables needed for appropriate piloting of policies vary considerably (Fay, 1996; Walker, 2001; Sanderson, 2002). Some measures such as changes in the school curriculum or campaigns to reduce heart disease may take years or even decades to produce a virtuous measurable effect. In these cases, the call for action tends to overwhelm the call for well-grounded prior evidence. Other policies are designed to have an almost immediate impact. In any event, most policies take time to bed in and the timetable for their policy trial needs to be 14

19 adjusted accordingly. Unless the period of the trial is long enough to detect certain impacts, it can create a false impression of policy failure which would have been contradicted by a later reading. There was a strong sense among the people we interviewed that these conflicts were not explicitly confronted when decisions to pilot or not to pilot were being made. Case Study 3 Drug Treatment and Testing Orders (DTTOs) Scottish Executive Aim: Pilot to inform decisions on whether to introduce DTTOs in Scotland and to provide evidence on the logistical, financial and crime-reduction implications of the policy. Background: DTTOs offer an alternative form of sentencing for dealing with drug users who commit crimes to fund their drug use, introduced in the Crime and Disorder Act Offenders have to participate in individually designed drug treatment programmes and submit to mandatory drug testing over the period of the order (lasting between six months and three years). The Home Office evaluated three pilot schemes in England and Wales and deemed them successful for roll-out. DTTOs were introduced in two schemes in Glasgow and Fife as a way of testing DTTOs in the local context. Methods : The evaluation studied the operation of the pilots and the effectiveness of DTTOs. The success of DTTOs was measured by their use among sentencing sheriffs (judges); by the success of the enforcement of orders; and by the retention of offenders in treatment programmes. Impact assessments were carried out on self-reported re-offending and offenders' spending on drugs. Treatment providers' and offenders' views on the effectiveness of DTTOs were also gathered. A variety of research methods was used, including the analysis of case files; observation of court reviews; questionnaire surveys among DTTO staff and treatment providers and in-depth interviews with stakeholders, including social work managers, DTTO staff, treatment providers, sheriffs, and offenders given DTTOs. A comparative cost analysis of DTTOs was also produced. Findings: DTTOs had an impact on reducing drug misuse and drug-related offending in the pilot areas. Multi-agency working was the biggest challenge faced by DTTOs and a lack of suitable treatment facilities available in some areas of the pilots was identified. Interim findings were, however, sufficiently encouraging that a phased national roll-out of DTTOs began in September Lessons learned: In developing the Scottish pilots, several lessons were learnt from the Home Office approach, resulting in an awareness-raising campaign among sheriffs in the run-up to the pilots and more effective methods of screening offenders. The pilots highlighted the interdependency of new policies with existing systems of provision and identified the need for developing a protocol for inter-agency working for the programme. The decision to phase-in DTTOs early was influenced by the fact that roll-out had occurred in England and Wales and by political pressures in the Scottish Parliament. The lead-in time for the pilot did, however, mean that the experience of some sentencers was limited at the time of the evaluation. Contact details/further information: Dr Joe Curran, Scottish Executive Criminal Justice Research Branch, 1W St Andrews House, Regent Road, Edinburgh, EH4 3DG, Tel: , Joe.Curran@scotland.gsi.gov.uk Eley, S., Gallop, K., McIvor, G., Morgan, K. and Yates, R. (2002), Drug Treatment and Testing Orders: Evaluation of the Scottish Pilots, Edinburgh: The Scottish Executive. The Research Report is available at The Research Findings Paper is available at 3. THE CASE FOR PILOTING 15

20 4. PILOT METHODOLOGIES Trying It Out The Role of Pilots in Policy-Making 4.1 Alternative approaches Even more tricky than decisions about whether to conduct pilots are decisions about how to conduct them. Methods of evaluating a new policy may be summative, formative or both. Summative methods are used to determine whether and to what extent a policy is having its desired effect or impact on its intended target groups. Formative methods are used to shape a policy and/or determine why, how or under what conditions it may be best directed or implemented. Both sorts of evaluations use a range of research methods but typically summative evaluations employ quantitative and/or experimental methods, while formative evaluations rely more on qualitative and/or ethnographic methods. But these distinctions are by no means rigid. These two broad approaches are complementary rather than competitive, in much the same way as there is a need for both quantitative and qualitative methodology in piloting as in all other forms of evaluation. What matters is rigour and fitness for purpose, not an a priori methodological preference. 4.2 Experimental methods randomised controlled trials (RCTs) Widely acknowledged as the most robust and rigorous of these approaches though sometimes ruled out on practical or political grounds is the randomised controlled trial (RCT), best known for its pivotal role in medical research. In its purest and simplest form, a random sample of people (or units such as schools, housing estates or hospitals) is selected for the experimental design. A random half of them are allocated to the treatment or experimental group, while the other half are allocated to the control or comparison group to measure the counterfactual. As long as the samples in each group are sufficiently large, differences in the outcomes of the two groups can reasonably be attributed to the treatment. The principle behind a randomised controlled trial is that other exogenous or confounding factors that might otherwise influence outcomes ought to be randomly distributed between the treatment and the control group. As noted, RCTs of individuals are a major form of policy-testing for social interventions in the US and Canada (Greenberg and Shroder, 1997; Boruch, 1997). In Britain, however, while they are still routinely employed in medical trials, they are much more sparingly applied in social policy interventions. Even so, a number of major British pilots have used RCTs, such as the Restart programme (White and Lakey, 1992), New Deal for 25+ (Wilkinson, forthcoming), Employment Zones, Intensive Gateway Trailblazers (Davies and Irving, 2000), and more recently the ERA project (Morris et al., 2003) (Case study 1, p.9). But most British social policy pilots tend to be conducted not only by means of area-based trials in preference to individual-based ones, but also by means of matched comparisons rather than random assignment. 16