A Comparative Study on Impact Measurement of. Social Impact Bonds for Social Service Innovation

Transcription

1 A Comparative Study on Impact Measurement of Social Impact Bonds for Social Service Innovation Hideaki BABA *, Ichiro TSUKAMOTO **, Takayuki YOSHIOKA *** Abstract Social Impact Bond (SIB) is well-known for its unique funding scheme, and its main structure is an outcome-based payment contract. To explore impact measurement used in SIBs, we conducted interviews with multiple SIB stakeholders in the United Kingdom. We found that in SIBs, performance is not only measured as impact by using objective methods, but also measured as outcomes through supplemental output indicators. We argue that evidence-based impact measurement is useful to innovate the process of social services provision, and continuous investigations are necessary to find reliable impact indicators with sufficient level of evidence. Key words: social impact bond, social service innovation, impact measurement, evidence-based evaluation, outcome-based contract * Professor, Faculty of Business and Commerce, Kansai University, Japan (baba-hid kansai-u.ac.jp) ** Professor, Department of Public Management, Meiji University, Japan *** Research fellow, Institute for Nonprofit and Public Management Studies, Meiji University, Japan 1

2 1. Introduction Social Impact Bond (SIB) is well-known for its unique funding scheme, and has already been implemented in several countries. According to Social Finance (2010), an outcome-based payment contract is the main structure of SIB s funding scheme, and simultaneously regulates methods of performance measurement of social services. Performance of social services is usually measured as outputs, outcomes or impacts. In general, outputs mean an amount of intervention provided by the program, and outcomes mean an intended achievement accomplished by such intervention. In contrast, impact refers to net outcome indicators as a proxy of impact in SIBs although the concept of impacts is frequently not clearly defined (The Guild, 2010). Figure 1 shows an example of impact measurement methods which uses the net outcome concept. Net outcome is measured as a change of outcome achieved by a certain program. As shown in Figure 1, when we compare two programs to reduce prisoners, program B seems to be more preferable because future jail days are shorter in absolute-basis. However, when we consider the impacts of the programs, program A is more preferable because the change of outcome achieved by program A is bigger. 2

3 Figure 1: Method of Impact Measurement Source: MDRC(2013:18) 2. Research method Evidence-based evaluation is the most outstanding characteristics for impact measurement. Social Impact Investment Taskforce (2014:23) describes generally accepted indicators as impact language to clarify impacts or outcomes and claims that supporting data infrastructure is essential for impact measurement. Big Society Capital also produces Social Outcome Matrix concerning nine social service areas 1. When supposing a youth employment support project, the outputs may be the number of youth who have completed job training programs, and the outcomes may be the number of persons who have obtained jobs after completion of the programs. However, the impacts require evaluation for broader and longer outcomes because if 1 Nine social service areas are employment, training and education; housing and local facilities; income and financial inclusion; physical health; mental health and well-being; family, friends and relationships; citizenship and community; arts, heritage, sports and faith; conservation of the natural environment. 3

4 this program has succeeded, welfare-payments will be reduced and income tax revenues will be increased. To improve performance-oriented social service provision, evidence-based impact measurement has been implemented in some countries To collect data for evidence-based evaluation, Social Impact Investment Taskforce (2014:19-20) requires five data qualifications such as materiality, reliability, comparability, additionality, and universality. In addition, New Philanthropy Capital (2014) developed four-pillar approach to decide a required level of evidence and to select data sources for evaluation. However, the impact measurement of SIBs is partly different from pervasive impact measurement methods such as social return on investment (SROI), because the impact measurement of SIBs often focuses on impacts or outcomes related to the reduction of government fiscal deficits. In addition, several SIBs have recently been introduced into social service programs for the reason that they are worth trying to improve public welfare by transferring financial risks from the government to private investors, although these SIBs might not fully achieve agreed outcomes in the contracts. To clarify such unique characteristics of SIBs impact measurement, we conducted interview with government executives, intermediaries, service providers, and social investors in the United Kingdom. These interview results are shown in Table 1. 4

5 Place Peterborough, England Greater London, England Essex County, England Whole of England Newcastle, England Interview The Young Foundation ( ) Social Finance ( , ) St Mungo's Broadway ( , ) Essex County Council ( ) Action for Children ( ) It's All About Me ( ) Ways to Wellness ( ) Table 1: Impact Measurement of SIB projects Term of project from 2010 to 2016 Amount of investment (canceled in 2015 due to 5 million government policy change) from 2012 to 2015 from 2013 to 2018 from 2013 to 2023 from 2015 to thousand 3.1 million 2 million 1.7 million Objective Provide a package of intensive support services for prisoners including housing assistance, drug and alcohol treatment, employment assistance, parenting assistance, and mental health support to reduce recidivism. Provide individual intervention plans to get people off the streets and into stable accommodation, thereby increasing prospects of employment or training and stabilizing health. Deliver family therapy in the home through highly qualified therapists over three to five months with the aim of keeping families together and avoiding out-of-home care. Assist the tough adoption of children who are racial minorities, have disabilities, or have siblings. Improve selfmanagement for patients with Long- Term Conditions (LTCs) and reduced costs of secondary healthcare services. Target 3,000 shortterm (sentences of less than 12 months) male prisoners aged 18 and older released from Peterborough Prison. Fixed cohort of 416 rough sleepers who were seen rough sleeping and/or stayed in a rough sleeping hostel in the last 3 months, and seen rough sleeping at least 6 times over the last 2 years. 380 youth allocated to 20 cohorts aged 11 to 16 year olds at the edge of out-of-home care or custody, and their families. Performance indices prescribing payments Reduction in the reoffending rate compared to the average of a control group of matched offenders over the 12 months following release from the Peterborough Prison. [impact] (1) Reduction of rough sleeping [outcome] (2) Accomodation for over 12 month [outcome] (3) Sustained reconnections to home countries [output] (4) Reduction in A&E visits [output] (5) Sustained volunteering, part-time or full-time work [output] (6) Job training [output] Comparison of historical 650 cases over a period of 30 months and the number of days spent in residential care for each family. [impact] 650 or more (1) Child enters program children in state registration [output] care likely ages (2) Child placed with 4 to 18, family [output] particularly (3) 1st anniversary of a waiting for an placement [outcome] adoptive family (4) 2nd anniversary of for over a year. placement [outcome] 8,500 patients with Long- Term Conditions (LTCs) in Newcastle West Clinical Commissioning Group (CCG). (1) Measure improvements of patient s health and wellbeing in 8 areas that relate to the ability to live with LTCs by using patient selfreporting metrics, Wellbeing Star. [output] (2) Measure the secondary care costs savings by comparing the control cohort in Newcastle North & East CCG. [impact] Fiscal deficit reduction Calculate probabilityweighted cost concerning police, court, and prison or community sentence. Estimate the cost which would be burdened by the local government if SIB project is not adopted. Compare the care cost provided by the local government and SIB project. Furure costs, which is avoidable if the adoption is succeeded, to preserve and raise children. Estimated future costs reduction of secondary healthcare services. 5

6 3. Results The performance of social services is generally measured by outputs and outcomes. Outputs are usually evaluated by measuring the quantity of provided services, and outcomes are often evaluated by measuring the quality of achievement. However, the both outputs and outcomes are evaluated by measuring quantitative indicators in SIBs. In this study, we define outputs as indicators which do not directly achieve program objectives but indirectly affect future impacts, and outcomes as those which directly relate to program objectives but are not evaluated by comparing a treatment group with a control group. In addition, we define impacts as the change of outcomes which is purely induced by the SIB s intervention. The Peterborough SIB project adopted a typical impact measurement method, namely measuring the reduction in the re-offending rate through quasi-experimental matched control group (propensity score matching). Furthermore, the Essex SIB project can be also classified as impact measurement because the project conducts historical comparison of 650 cases. In contrast, St Mungo s Broadway s Street Impact SIB project does not conduct a comparison between target group and control group, and thus, the outcome evaluation method is not an impact measurement method. However, the target cohort of 6

7 416 rough sleepers was selected at the beginning of the SIB project, and, thus, distinct outcomes would be measured by monitoring the persons who do not require the intervention any longer. In addition, the output indicators are also adopted in the SIB project. Reconnections to home countries, reduction in accident and emergency visits, intermediate labour, and job training are not the main purposes of the SIB project, but are expected to improve individual wellbeing in the future. Similarly, output indicators are used for the It s All About Me (IAAM) SIB project, a children adoption program in England. The ultimate objectives of this SIB project are to keep good relationships between adopted children and adoption families, and ultimately individual wellbeing of adopted children. However, the number of registration for adoption and placement with adoption family are also employed as the output indicators to achieve the ultimate objectives. Moreover, in the Ways to Wellness SIB project, Newcastle West Clinical Commissioning Group (CCG) adopted more obscure output indicators. This SIB project evaluates its performance by measuring self-reporting metrics, which is considered to improve the condition of patients with long-term diseases 2. These output indicators are related to the enhancement of health promotion according to some academic research. 2 The eight area of self-reporting metrics consists of lifestyle, feeling positive, work, volunteering and other activities, family and friends, managing symptoms, where you live, and looking after yourself. 7

8 The secondary performance is measured as impacts based on the savings on health care costs by comparing the health care costs in Newcastle West CCG with those of Newcastle North and East CCG. 4. Conclusion Theoretically, the performance of SIBs should be measured as impacts by applying an objective method such as Randomized Controlled Trial (RCT). However, RCT requires high costs and efforts to set comparative groups. Thus, from a practical viewpoint, outcome indicators are also applicable and useful by properly employing supplemental output indicators. The principal objective of SIBs impact measurement is to evaluate performance of social services and to use such results to reduce government spending. Investors and evaluators regularly require data on the impacts, outcomes and outputs of SIB interventions. Service providers and intermediaries we interviewed often mentioned that evidence-based impact measurement is useful to innovate the process of social service provision. Thus, performance-based management seems to be appreciated by service providers. We found that impact measurement methods applied to SIBs are diverse and 8

9 flexible, and unproved indicators are adopted to induce innovative changes of social services. Continuous investigations are necessary to explore reliable impact and outcome indicators which sufficient level of evidence. Acknowledgment: This research was supported by JSPS KAKENHI Grant Number (16K04021) and MEXT-Supported Program for the Strategic Research Foundation at Private Universities ( ). 9

10 References MDRC (2013) Financing promising evidence-based programs: early lessons from the New York City Social Impact Bond, grams_fr.pdf (2015/11/16). New Philanthropy Capital (2014) Building your measurement framework: NPC s four pillar approach, (2015/7/23). Social Finance (2010) Towards a new social economy: Blended value creation through Social Impact Bonds. Social Impact Investment Taskforce (2014) Measuring impact, Impact WG paper FINAL.pdf (2015/7/23). The Guild (2010) Getting Started in Social Impact Measurement, cial_impact_measurement_-_ pdf (2015/9/12). 10