MEASURING AND/OR ESTIMATING SOCIAL VALUE CREATION: Insights Into Eight Integrated Cost Approaches

Transcription

1 MEASURING AND/OR ESTIMATING SOCIAL VALUE CREATION: Insights Int Eight Integrated Cst Appraches Prepared fr Bill & Melinda Gates Fundatin Impact Planning and Imprvement Prepared by Melinda T. Tuan P.O. Bx 805 Narberth, PA FINAL PAPER December 15, 2008

2 TABLE OF CONTENTS 1. OVERVIEW Intrductin 1.2. Purpse 1.3. Methdlgy 1.4. A Wrd n Language 2. CURRENT STATE OF INTEGRATED COST APPROACHES TO MEASURING AND/OR ESTIMATING SOCIAL VALUE Lack f Maturity in Scial Prgram Evaluatin 2.2. A Variety f Purpses 2.3. N Silver Bullet(s) 3. PROFILES OF EIGHT INTEGRATED COST APPROACHES TO MEASURING AND/OR ESTIMATING SOCIAL VALUE Cst-Effectiveness Analysis (CEA) 3.2. Cst-Benefit Analysis (CBA) 3.3. REDF Scial Return n Investment (SROI) 3.4. Rbin Hd Fundatin Benefit-Cst Rati 3.5. Acumen Fund BACO Rati 3.6. William and Flra Hewlett Fundatin Expected Return 3.7. Center fr High Impact Philanthrpy Cst per Impact 3.8. Fundatin Investment Bubble Chart 3.9. Summary f Methdlgies 4. TECHNICAL ISSUES AND LIMITATIONS Assumptins 4.2. Discunt Rates 4.3. Timeframes 4.4. Shadw Prices 4.5. Interdependencies 4.6. Value Judgments 5. BIG PICTURE ISSUES Incnsistent Use f Language 5.2. Lack f Cmmn Measures in the Scial Sectr 5.3. Lack f Quality Data n Impacts, Outcmes, Outputs, and Csts 5.4. Lack f Incentives fr Transparency 5.5. Unintended Cnsequences 5.6. Inadequate Utilizatin 5.7. Cst f Measurement 6. SUMMARY OF ISSUES AND IMPLICATIONS FOR THE SOCIAL SECTOR AUTHORSHIP AND RESEARCH CREDITS Melinda T. Tuan ii

3 TABLE OF CONTENTS (cntinued) APPENDICES A. Glssary f Terms 27 B. Leading Practitiner and Expert Interview List 34 C. Measuring and/r Estimating Scial Value Meeting Participant List.. 35 D. Bibligraphy.38 E. Applicatins f Integrated Cst Appraches t Measuring and/r Estimating Scial Value F. Cnceptual Basis fr Integrated Cst Appraches t Measuring and/r Estimating Scial Value G. Cst-Effectiveness Analysis H. Cst-Benefit Analysis I. REDF SROI J. Rbin Hd Fundatin Benefit-Cst Rati K. Acumen Fund BACO Rati L. Hewlett Fundatin Expected Return M. Center fr High Impact Philanthrpy Cst per Impact N. Fundatin Investment Bubble Chart O. The Flat f the Curve 43 Melinda T. Tuan iii

4 1. OVERVIEW 1.1. Intrductin In the field f philanthrpy, there is currently a great deal f enthusiasm fr applying business principles and investment analyses t decisins abut funding nnprfit rganizatins and prgrams. As Lynn A. Karly, Ph.D., Senir Ecnmist with the RAND Crpratin states, the discipline assciated with these hard-nsed business management appraches is perceived t be a useful antidte t the ften emtinal appeals that accmpany funding decisins in philanthrpy and plicy discussins and decisin-making in the public plicy arena. 1 These appraches all integrate measures f cst in their calculatins f the relative benefits f funding a particular prgram r rganizatin. As the scial sectr cnsiders whether and hw t mre systematically integrate cst int measuring the scial impact f its philanthrpy, it may be helpful t d three things: 1) Identify and review the leading and prmising appraches t incrprating cst int measuring and/r estimating scial value in the scial sectr 2) Analyze the relative strengths and weaknesses f these appraches and identify any crss-cutting issues 3) Understand hw these examples and insights can infrm the scial sectr s explratins abut the csts f achieving scial impact T these ends, this paper will describe and analyze eight appraches t integrating cst in measuring and/r estimating scial value creatin. These include tw classical methdlgies (cst-effectiveness analysis and cst-benefit analysis) and six prmising appraches that have been develped by philanthrpic and nnprfit rganizatins in the last decade. Mst f these new methdlgies draw n cncepts frm cst-effectiveness and cst-benefit analysis. The purpse f prfiling and analyzing these appraches is nt t chse the right ne, fr there is n perfect methdlgy. Rather, it is t present sme fresh pssibilities fr thinking abut the cst-benefit f philanthrpic investments. These varius appraches prvide different lenses fr viewing scial value creatin and bring a new level f rigr and creativity t the measurement r estimatin f scial value. They als illustrate the hst f limitatins related t effrts t measure and/r estimate scial value, bth technical and big picture issues, mst f which are crsscutting issues fr the field. The implicatins f these pssibilities and limitatins will serve as a reference pint fr thse in the scial sectr wh are cnsidering whether and hw t craft their wn appraches t integrating cst int their scial impact measurement effrts. Additinally, these implicatins will help clarify whether it is pssible t pursue a methdlgy that can be adpted acrss the sectr. 1 Karly, Lynn A., M. Rebecca Kilburn, James H. Bigelw, Jnathan P. Caulkins, Jill S. Cannn. Chapter Tw: Overview f Cst and Outcme Analysis. RAND: Assessing Csts and Benefits f Early Childhd Interventin Prgrams: Overview and Applicatins t the Starting Early, Starting Smart Prgram. p. 5. Melinda T. Tuan 4

5 1.2. Purpse This paper was cmmissined by Impact Planning and Imprvement (IPI) t take a first lk at sme f the leading examples f integrated cst appraches t measuring and/r estimating scial value in the scial sectr. The paper and the cmpanin appendices are structured t prvide the larger cntext fr and generate further discussin amng philanthrpic and nnprfit leaders in the scial sectr regarding the current effrts and future plans thrughut the sectr t integrate cst int scial impact measurement activities. T aid in this effrt, the paper prvides a cmmn language (Appendix A) and detailed examples f the varius methdlgies fr leading practitiners t reference as they cnsider whether, where, and hw t g deeper in understanding these issues and the implicatins fr their rganizatins and the sectr as a whle. This paper des nt represent a cmprehensive scan f all the integrated cst effrts in the nnprfit and philanthrpic sectrs. This paper als des nt review the mre extensive histrical and current uses f cst-effectiveness analysis and cst-benefit analysis t measure impact in the gvernment sectr. This paper des, hwever, illustrate sme different ways cst is being integrated int a variety f measurement framewrks that are currently being used r cntemplated by leading philanthrpic and nnprfit rganizatins. The paper includes a high level view f eight different methdlgies, the technical limitatins and big picture issues represented by these cllective methdlgies, and the implicatins fr the scial sectr. A lengthy and detailed set f appendices cvers each f the eight methdlgies with a higher degree f granularity, includes an example f hw each methdlgy is applied, and als discusses each appraches benefits, limitatins, and utilizatin Methdlgy This paper is based n interviews with leading practitiners and experts in measuring and/r estimating scial value (Appendix B); a meeting f leading practitiners, experts, funders and staff frm the Bill & Melinda Gates Fundatin held in Octber, 2008 t discuss a draft versin f this paper (Appendix C); a brief literature review f cst-effectiveness analysis and cst-benefit analysis and materials abut the six prfiled rganizatins and appraches (Appendix D); and nging cnversatins with Fay Twersky and Kendall Guthrie f Impact Planning and Imprvement A Wrd n Language The rganizatins we prfiled in this paper ften use different wrds t describe the same thing r use the same wrd t describe different things. This can be very cnfusing and bfuscate the true methdlgies r results behind the varius appraches. T prvide greater clarity, we refer t Appendix A: Glssary f Terms t define the many technical terms used in this paper. In additin, we will use the term scial value creatin r scial value thrughut the paper t refer t the general cncept and practice f measuring scial impacts, utcmes, and utputs thrugh the lens f cst. When apprpriate, we will ftnte terms that are defined differently in the varius methdlgies fr measuring and/r estimating scial value and explain, t the best f ur knwledge, what they really mean. Melinda T. Tuan 5

6 2. CURRENT STATE OF INTEGRATED COST APPROACHES TO MEASURING AND/OR ESTIMATING SOCIAL VALUE Based n interviews with experts, leading practitiners, and a scan f the literature, integrated cst appraches t measuring and/r estimating scial value in the scial sectr have nt yet reached maturity. This is due in large part t the lack f maturity in scial prgram evaluatin methdlgies and the variety f purpses rganizatins have fr cnducting these types f analyses Lack f maturity in scial prgram evaluatin In 2007, the MacArthur Fundatin cmmissined the RAND Crpratin t cnduct a study f 39 effective scial prgrams that have been evaluated using scientifically rigrus methds. Of these, 22 scial prgrams had been the subject f ne r mre cst-benefit analyses. The purpse f the study was t examine the state f the field f valuing benefits in scial prgrams. Dr. Lynn Karly, Senir Ecnmist at RAND, published a paper in Nvember 2008 which cncluded that the applicatin f the cst-benefit methdlgy in evaluatins f scial prgrams has nt reached maturity. 2 Karly s findings which led her t this cnclusin included factrs such as: Many imprtant benefits that accrue frm effective scial prgrams are rarely, if ever, mnetized Shadw prices (the dllar values assigned t utcmes) in cst-benefit analyses f scial prgrams d nt cnsistently capture the full range f scietal benefits r csts Even when there is well-established literature fr valuing utcmes, shadw prices are nt being cnsistently used acrss studies f scial prgrams Sme cst-benefit analyses use methds t prject future utcmes based n early utcmes, but such appraches have yet t becme rutine and standardized Overall, these limitatins pint t the fact that the field f scial prgram evaluatin the prcess f cllecting scial impact and scial utcme data and the methds f calculating the csts f scial prgram delivery are nt very well develped r established in the scial sectr. Despite these limitatins, sme peple expect t be able t cmpare the scial value f varius scial prgrams similar t hw they cmpare the financial return n investment (ROI) f varius cmpanies. This is nt a reasnable r realistic expectatin given that the infrastructure necessary t calculate scial value creatin fr scial prgrams is virtually nn-existent. The infrastructure that makes financial ROI calculatins pssible (e.g. the accunting prfessin, brkers, financial analysts, financial reprting, financial cncept develpment), has taken a lng perid f time (sme might argue centuries) t develp and there are still cnstant debates abut hw ecnmic value is measured and hw much value cmpanies are creating. 2 Karly, Lynn A. Valuing Benefits in Benefit-Cst Studies f Scial Prgrams. RAND, Melinda T. Tuan 6

7 In cmparisn, the scial sectr has really nly begun t measure scial utcmes in the last few decades. While there are a handful f grups such as RAND and MDRC that cnduct cst-effectiveness analyses and cst-benefit analyses and a few SROI cnsultants, there is n scial auditing prfessin that des these types f analyses in a unifrm manner fr the scial sectr. Until a tremendus amunt f resurces are invested in creating a cmparable infrastructure fr measuring and analyzing the results fr the scial sectr, integrated cst appraches t measuring and/r estimating scial value will cntinue t be practiced mre like an islated art frm than widespread science A Variety f Purpses There appears t be little cnsensus amng leading practitiners f integrated cst appraches t measuring and/r estimating scial value abut hw ne shuld use cst-related impact data t make certain investment decisins. In general, there are fur philsphical psitins behind the methdlgies and rganizatins self-identify in ne r mre f the different categries: 1) One can and shuld use cst and impact data t make funding allcatin decisins acrss prgram areas Michael Weinstein, Chief Prgram Officer at Rbin Hd Fundatin is emphatic in stating his case: It s impssible nt t d it. If yu re making grants, yu re placing yur bets yu are assigning implicit values t the activities that yu fund. Sme envirnmentalists dn t like the idea f deciding hw many snail darters equal the value f a plar bear, even thugh they ve made that decisin implicitly nce their rganizatins set their annual budgets. There is, hwever, virtue in being explicit making decisins with analysis afrethught rather than relying n utcmes dictated by an amrphus prcess that desn t face up t tradeffs. Perhaps we can agree that spending mney t save three snail darters at the cst f freging a prgram that wuld save five plar bears wuld be ridiculus. But shuld we spend mney t spare the extinctin f snail darters at the cst f lsing half the ppulatin f plar bears? Better t face tradeffs explicitly than t behave passively, implicitly. 3 Paul Brest, President f the William and Flra Hewlett Fundatin shared his perspective: I think yu can nly d it when yu have a cmmn utcme. Our six prgrams range frm perfrming arts t envirnment t glbal develpment. In cntrast, Rbin Hd s different prgrams are all cncerned with alleviating pverty in New Yrk. But even when yu have a cmmn utcme, it s a bit tenuus as the margins f errr are huge. The interesting questin frm all this is: Why d we think it s imprtant? 4 2) One can nly use cst and impact data t make funding allcatin decisins within prgram areas Brian Trelstad, Chief Investment Officer fr Acumen Fund stated, I think nce yu ve chsen an area that matters t yu then yu shuld play this game. But there shuld be limits. Yu can cmpare prgrams nce yu get in the sectr f glbal 3 Weinstein, Michael. Re: Rbin Hd Fundatin Benefit-Cst Rati and Integrated Cst Appraches t Measuring Scial Impact t Melinda Tuan, 29 July Brest, Paul. Re: Hewlett Fundatin s Expected Return Methdlgy t Melinda Tuan, 18 July Melinda T. Tuan 7

8 health, but yu can t cmpare glbal health vs. arts vs. climate change. Yu can t bil it dwn t that. It s flish t say ne climate change initiative is wrth mre than tw educatin initiatives. 5 3) One can ideally use cst and impact data t making funding decisins acrss prgram areas but in reality, ne wuld be lucky t have access t cst and impact data t make funding allcatin decisins within a prgram area at all nt t mentin assessing whether a prgram in and f itself is a wrthwhile investment Kat Rsqueta, Executive Directr f the Center fr High Impact Philanthrpy explained their apprach: We re nt lking fr a unifying measurement acrss dmains We re develping a cnceptual framewrk f having the biggest impact acrss a dllar unit. We re nt trying t d this acrss glbal health and urban educatin. In fact, we re nt even trying t use the same unit within urban educatin. Given the lack f gd, empirical infrmatin t build n, the required assumptins and mdeling wuld make the results academic. 6 4) One shuld use cst and impact data t prmte the wrk f individual nnprfit rganizatins and prmte the use f scial value creatin metrics t build the field f scial prgram evaluatin Jeremy Nichlls, Fellw at New Ecnmics Fundatin (nef) described his purpse fr develping and prmting the scial return n investment (SROI) methdlgy: Many rganizatins with scial bjectives were nt reprting n the relatinship between their investment and the utcmes they were achieving. What we want is a cnsistent apprach t measuring value: get rganizatins t frecast scial returns, build the systems t track thse ver time, then lk back and see hw thse went. A turning pint wuld be if we culd get investrs and funders interested where their funding criteria included using SROI principles. Then we culd get t a level where there will be enugh cmmnality f measures that there will be cmparability within areas. Acrss these fur philsphical perspectives, there is anther way t categrize the purpses f measuring and/r estimating scial value using the element f time. The three primary applicatins are: 1. Prspective lking frward t pssible philanthrpic investments t determine whether r nt the prjected csts and benefits in the future indicate a favrable investment in the present 2. Onging testing assumptins and prjectins regarding intended scial value creatin alng the way, in rder t aid in curse crrectin 3. Retrspective lking back at past philanthrpic investments t determine whether r nt they were favrable investments given the csts incurred, in rder t infrm future decisins 5 Trelstad, Brian. Re: Acumen Fund Best Available Charitable Optins (BACO) and Prtfli Data Management System (PDMS) t Melinda Tuan, 30 April Rsqueta, Kat, Hilary Rhdes, and Kathleen Nnan. Re: Center fr High Impact Philanthrpy Cst Per Impact Measures and ther Integrated Cst Appraches t Measuring Scial Impact t Melinda Tuan, 8 May Melinda T. Tuan 8

9 As with the previus set f purpses, the varius appraches may incrprate ne r mre f these pssible applicatins (Appendix E and Appendix F) N Silver Bullet(s) It is imprtant t cnsider each f these methdlgies in the greater cntext in which the rganizatin makes decisins. Regardless f the specific purpse, each methdlgy and its accmpanying results are nly ne factr in an rganizatin s decisin-making prcess. Jed Emersn, Funding Directr f REDF, explained: At REDF we went t great lengths t create a set f analyses s SROI wasn t biled dwn t ne number. We specifically tried t avid the trap f cming up with a single numeric. 7 Susan Stut, recently retired Manager f the Wrld Bank s Results Secretariat cautined: There is incredible silver bulletism arund in the dnr (and perhaps fundatin) wrlds seeking that ne special number that will tell us if we are succeeding r failing. This is driven by bureaucratic fantasy, nt reality. The chances that we culd cme up with a metric that avids an inevitably subjective prcess f judgment and chice are infinitely small (else plitics wuld be a much simpler and bring tpic). It s usually driven by a desire t define a bttm line that will d fr philanthrpy and public sectr management what prfit/lss statements d fr the private sectr. It s just nt ging t happen that way Emersn, Jed. Re: Integrated Cst Appraches t Measuring Scial Impact and REDF s SROI t Melinda Tuan, 1 May Nte: Despite REDF s desires t avid the trap f a single numeric, many f the subsequent cst integrated measurement methdlgies based their appraches n the single SROI metric f the blended value index f return. Stut, Susan. Re: Perspectives n Cst-Benefit Analysis and Cst-Effectiveness Analysis in Glbal Health cmmunicatin t Philip Setel, April Melinda T. Tuan 9

10 3. PROFILES OF EIGHT INTEGRATED COST APPROACHES TO MEASURING AND/OR ESTIMATING SOCIAL VALUE The fllwing is a high level verview f eight different appraches t integrating cst int measuring and/r estimating scial value creatin. The first tw are classical appraches, including cst-effectiveness analysis (CEA) and cst-benefit analysis (CBA). The next six appraches represent leading practitiner methdlgies, including bth philanthrpic and nnprfit rganizatins Cst-Effectiveness Analysis (CEA) (Appendix G) Cst-Effectiveness Analysis (CEA) invlves the calculatin f a rati f cst t a nnmnetary benefit r utcme (e.g. cst per high schl graduate, cst per child cured f malaria). This rati is smetimes infrmally termed the bang fr the buck. CEA is used in situatins when mnetizing the benefits f a prgram r interventin is nt pssible r apprpriate. Hwever, measures f cst-effectiveness can nly accunt fr ne area f prgram impact at a time. And, since prgram impacts are measured in natural units (e.g. life year saved, child graduating frm high schl), unless thse units are cmmn acrss all areas f impact, it is nt pssible t aggregate acrss them. 9 The purpse f CEA is tw-fld: 1) t cmbine apprpriate measures f utcmes with csts s that prgram and plicy alternatives within the same dmain can be ranked accrding t their effectiveness relative t their results; and 2) t side-step the uncertainties abut hw t value different aspects f prgram benefits by lking at the rati f benefits t csts withut reducing them t cmmn units (e.g. mnetary units). CEA is used in the private, public, and nnprfit sectrs and is widely used in health care where csts f interventin are cmpared t their impact n an individual s quality-adjusted life years (QALYs) r disability-adjusted life years (DALYs) Cst-Benefit Analysis (CBA) (Appendix H) Cst-benefit analysis (CBA) mnetizes the benefits and csts assciated with an interventin and then cmpares them t see which ne is greater. CBA is the mst demanding apprach t analyzing csts and utcmes as it requires a cmprehensive measurement f csts and prgram impacts (e.g. primary and secndary, direct and indirect, tangible and intangible impacts), and the ability t place a dllar value n prgram impacts acrss stakehlders. Thus, CBA prvides a full accunting f the net benefits t sciety as a whle, as well as varius stakehlders. The purpse f CBA is twfld: 1) t help decide whether a prgram r interventin is f value t the decisin-maker and 2) t cmpare the prgram t alternatives and chse the ne with the greatest measure f merit. The utput frm cst-benefit analysis can be measures f net benefits (benefits csts) als knwn as the net present value (NPV); the rati f benefits t cst (benefit-cst ratis); r the internal rate f return (IRR) which is the rate f grwth a prject is expected t generate Karly, Lynn A. Valuing Benefits in Benefit-Cst Studies f Scial Prgrams. RAND, 2008, p Ibid. Melinda T. Tuan 10

11 CBA is widely used acrss the public, private, and increasingly the nnprfit sectr t help decisin-makers priritize r decide amng varius uses f funds fr prgrams and prjects REDF SROI (Appendix I) REDF is a nnprfit philanthrpic scial venture fund funded in 1997 in San Francisc, CA. REDF supprts emplyment fr lw-incme and frmerly hmeless individuals by making grants t a prtfli f nnprfit rganizatins in the San Francisc Bay Area that fully wn and perate varius scial enterprises. REDF develped its SROI framewrk in the late 1990 s culminating in the publicatin f the SROI Reprts and several SROI methdlgy dcuments and tls in The purpse f REDF s SROI was t demnstrate the scial, enterprise, and blended value accrued t sciety cmpared t the ttal investments fr each f the scial enterprises in its prtfli n an nging and retrspective basis. Since 2000, REDF has nt released any further SROI reprts. It is instead fcusing n cntinuing t measure and reprt n the scial utcmes f the enterprises in terms f individuals changed lives withut mnetizing the utcmes r cmparing these t their assciated csts. Even thugh REDF is n lnger implementing its SROI methdlgy, the cncepts underpinning REDF s SROI framewrk greatly influenced many f the appraches that have evlved in the past decade. A grup f internatinal practitiners including Jed Emersn, principals at new ecnmics fundatin (nef), Schlten & Franssen, SVT Grup, and thers published a revised apprach t calculating SROI in This revised apprach integrated REDF s SROI methdlgy with steps in cst effectiveness analysis and several ther methdlgies int an verarching SROI framewrk. This SROI Framewrk was updated in a bk published in Majr pints that differentiate the evlving methdlgy frm that f REDF include: applying SROI t any type f rganizatin r cmpany in any industry; accunting fr scial and envirnmental value created fr individual stakehlders; including stakehlder analyses; using shrter timeframes (5 years); and adjusting results fr the interdependencies f utcmes attributable t a set f rganizatins. An internatinal, decentralized netwrk including tw relatively new rganizatins, SROI UK and the Eurpean SROI Netwrk (ESROIN), cntinues t refine the SROI methdlgy and advcate fr its use thrughut Eurpe, the United States, and Suth and Sutheast Asia Rbin Hd Fundatin (Rbin Hd) Benefit-Cst Rati (Appendix J) Rbin Hd is a nnprfit funded in 1988 t target pverty in New Yrk City (NYC). Rbin Hd prvides nging grants t ver 200 NYC-based nnprfit rganizatins that fight pverty in fur general areas: Jbs & Ecnmic Security; Educatin; Early Childhd & Yuth; and Survival. Rbin Hd develped its Benefit-Cst Rati methdlgy in 2003 t capture the best estimate f the cllective benefit t pr individuals that Rbin Hd grants create per dllar cst t Rbin Hd (measured in part by the bst in incme f pr individuals due t the grant). 11 Schlten, Peter, Jeremy Nichlls, Sara Olsen, Brett Galimidi. Scial Return n Investment: A Guide t SROI Analysis. Lenthe Publishers, Melinda T. Tuan 11

12 The purpse f Rbin Hd s Benefit-Cst Rati is t translate the utcmes f diverse prgrams int a single, mnetized value that measures pverty fighting n an nging basis t answer the questin: Which prgrams t fund and hw much t spend n each. 12 Rbin Hd prgram fficers are required t calculate Benefit-Cst Ratis fr all f their new and renewal prgram funding prpsals n an annual basis (i.e., the which prgrams questin). Rbin Hd des nt, hwever, use the Benefit-Cst Ratis t make allcatin decisins amng prtflis decisins are made abut individual grants regardless f the prtfli f which they are a part. The distributin f spending acrss prtflis fllws as a passive cnsequence f decisins abut individual grants Acumen Fund (Acumen) BACO Rati (Appendix K) Acumen Fund is a nnprfit glbal venture fund funded in 2001 in New Yrk City. Acumen prvides capital investments ranging frm $300,000 t $2,000,000 in primarily debt r equity t a variety f institutins including nnprfit rganizatins and small, medium and large cmpanies. These funds supprt business mdels with a payback r exit in rughly five t seven years that can be effective in reaching the base f the pyramid (BOP) r the billins f pr. Acumen invests glbally in fur areas: Water; Health; Husing; and Energy. Acumen develped its Best Available Charitable Optin (BACO) Rati methdlgy in 2004 t quantify a ptential investment s scial utput 14 and cmpare it t the universe f existing charitable ptins fr that explicit scial issue. The purpse f the BACO Rati is t help prtfli managers assess the prspective merit f an individual investment pprtunity versus making a charitable grant. Ideally, the BACO Rati is re-assessed n an annual basis pst-investment. T date, Acumen prtfli managers have calculated BACO Ratis fr all f their prtfli cmpanies (sme retrspectively, sme as part f the due diligence prcess) and there are abut 25 active investments. Only a few annual BACO re-assessments have been calculated t date William and Flra Hewlett Fundatin (Hewlett) Expected Return (Appendix L) The William and Flra Hewlett Fundatin was funded in 1966 t slve scial and envirnmental prblems at hme and arund the wrld. Hewlett cncentrates its glbal grantmaking n six majr areas: Educatin; Envirnment; Glbal Develpment; Perfrming Arts; Philanthrpy; and Ppulatin. Hewlett develped its Expected Return (ER) methdlgy in 2007 t evaluate ptential charitable investments thrugh a systematic, cnsistent, quantitative prcess in rder t make every dllar cunt. The purpse f ER is t help Hewlett prgram fficers ask and answer the right questins fr every investment prtfli. Expected Return frces prgram fficers t test their implicit assumptins and thery f change/lgic mdel against the ER number, quantify high-level tradeffs between investments within an investment prtfli, and ideally make better prspective funding decisins within their 12 Weinstein, Michael. Re: Rbin Hd Fundatin Benefit-Cst Rati t Melinda Tuan, 29 July Ibid. 14 Acumen Fund s BACO methdlgy uses the term scial impact t describe scial utputs (e.g. they describe persn years f malaria prtectin as a scial impact ). Melinda T. Tuan 12

13 investment prtflis. T date, Hewlett has nly used ER in its glbal develpment area t make grantmaking decisins but plans t use it in ther areas Center fr High Impact Philanthrpy (CHIP) Cst per Impact (Appendix M) The Center fr High Impact Philanthrpy was established in 2006 by alumni f The Whartn Schl f the University f Pennsylvania (UPenn) wh were frustrated by the difficulty f measuring and maximizing the impact f their charitable gifts. Based ut f the Schl f Scial Plicy & Practice at UPenn, CHIP is a resurce center designed t guide philanthrpists and their advisrs as they decide where t allcate their philanthrpic dllars. Since 2006, CHIP has been develping its Cst per Impact methdlgy and intends t prmte it as a measure critical t high impact giving. CHIP is currently wrking n its first f several philanthrpic sectr reprts. These reprts analyze pprtunities fr individual philanthrpists t have impact and prvide exemplary case examples with assciated cst per impact estimates. The purpse f Cst per Impact is t prvide philanthrpists an answer t the questin, Hw much des change cst? 3.8. Fundatin Investment Bubble Chart (Appendix N) Sme nnprfits and fundatins are using a bubble chart t display cmparative infrmatin regarding multiple rganizatins. The purpse f the bubble chart is t illustrate a set f reprting metrics at the rganizatinal r prgram level that are cmmn acrss the prgrams f a nnprfit r a segment f a fundatin prtfli. Sample measures include number f peple reached with bed nets vs. percentage f bed nets utilized. The bubble chart allws ne t assess the individual and relative perfrmance f prgrams r rganizatins cmpared t the prgram size r fundatin investment at a single pint in time Summary f All Methdlgies In reviewing these different methdlgies fr measuring and/r estimating scial value creatin, Paul Brest, President f the William and Flra Hewlett Fundatin psited that in essence, all the methdlgies are abut expected return: Expected Return = (Outcme r Benefit X Prbability f Success) Cst Mst funders assume their prbability f success; then chse t either mnetize the benefits (CBA) r nt (CEA). A few funders als discunt the numeratr by the prprtin f their philanthrpic cntributin. As a heuristic, Brest s simplificatin f all the integrated cst methdlgies is helpful. Hwever, the eight different appraches have differently nuanced answers t these questins: Hw are the utcmes r benefits estimated? (e.g. randmized cntrl experiments, utputs used as prxies fr utcmes, timeframes, etc.) 15 Hugget, Jn. The Bridgespan Grup presentatin Business Planning: What it Is and Why it Matters p. 7; Olsen, Sara. Re: Integrated Cst Appraches t Measuring Scial Impact t Melinda Tuan, 30 April; Fay Twersky, Brian Ellit, Melinda Tuan interpretatin. Melinda T. Tuan 13

14 Hw are the csts calculated? (e.g. cst f grants and administratin, cst f ttal prgram, etc.) Hw are uncertainties and partial attributin f results accunted fr? (e.g. prbability f success, philanthrpic cntributin, interdependencies, etc.) Hw are the utcmes r benefits translated int natural units r mnetized? (e.g. shadw prices, discunt rates, etc.) At this pint in time, n single methdlgy has been widely adpted thrughut the scial sectr. And it is premature t declare that ne right methd fr measuring and/r estimating scial value creatin shuld be prmted. Hwever, fr rganizatins that are interested in integrating a cst apprach t measuring scial impact, there is significant value in emplying a single, cnsistent methdlgy thrughut the rganizatin. Philanthrpic rganizatins and nnprfit practitiners are finding the discipline inherent in the prcess f measuring and/r estimating scial value creatin imprves their wn practice and helps them fcus n the best methds t achieve their scial missin. Melinda T. Tuan 14

15 4. TECHNICAL ISSUES AND LIMITATIONS In reviewing the eight different methdlgies, we identified a number f technical issues that cut acrss many f the appraches. These issues include the use f: Assumptins Discunt rates Timeframes Shadw prices Interdependencies Value judgments These technical issues represent limitatins f the tls used t integrate cst appraches fr measuring and/r estimating scial value Assumptins There are a multitude f assumptins invlved in all f these methdlgies and calculatins. Sme assumptins, such as the Rbin Hd Factr r Hewlett s philanthrpy s cntributin cannt be easily tested r measured. Other assumptins, including prjectins f utputs r utcmes, r applicatins f expert research t a similar prgram s utcmes, can be measured retrspectively fr their accuracy. Hwever, this des nt seem t be dne n a cnsistent basis in any f the examples. In respnse t ne f the skepticisms f Hewlett s Expected Return methdlgy that yu re putting in a lt f incredibly speculative numbers, Brest argues: But ding the numbers presses prgram fficers t test their intuitins, and that s likely t sharpen them. 16 Weinstein f Rbin Hd Fundatin acknwledges that there s n way t get arund the ugly prblem. The virtue f ur metrics is that they are brutally clear abut the assumptins we re making alng the way, sme f them embarrassing. 17 In examining sme f the assumptins and their applicatin in sensitivity r scenari analyses, it is clear that several rganizatins are verly ptimistic in their prjectins. This ptimism includes prjectins f scial utputs, scial utcmes and impacts, prjectins f financial perfrmance, and the timeframe fr achieving these results. A few f the rganizatins methdlgies have been in existence fr lng enugh t cnfirm that the actual scial value created frm specific individual investments was significantly less than the riginally prjectins Discunt rates In rder t calculate the cst-effectiveness r cst-benefit f an interventin which results in benefits in future years, ne must discunt thse benefits t reflect the time value f mney. Hwever, while there is cnsensus that future utcmes shuld be discunted, there is n cnsensus as t what rate shuld be used says 16 Brest, Paul. Re: Hewlett Fundatin s Expected Return Methdlgy t Melinda Tuan, 24 April Weinstein, Michael. Re: Rbin Hd Fundatin Benefit-Cst Rati and Integrated Cst Appraches t Measuring Scial Impact t Melinda Tuan, 23 April Melinda T. Tuan 15

16 Karly. 18 Karly adds that while 4 percent is typical in the drug, criminal justice, and children and yuth interventin plicy areas, ther fields use different discunt rates. In medicine, discunt rates f 3 t 5 percent are recmmended and in ther fields discunt rates f 10 percent r higher have been used. The chice f rate may be a functin f the time preference f the stakehlder r decisin-maker 19 adding t the lack f cnsistency in hw csts and benefits are calculated acrss the field Timeframes There are a variety f timeframes being used thrughut these methdlgies. Fr example, Karly fund that sme prgrams that serve children and yuth d nt track their participants int the future while thers d. As a result, the amunt f actual utcme data available fr analysis varies widely acrss prgrams. 20 There is als little cnsistency in the use f timeframes fr analysis, regardless f whether there is actual impact r utcme data. REDF used a ten-year hrizn fr calculating benefits, Rbin Hd a range up t 30 years, nef 5 years. All f these variatins n timeframe cntribute t very different results that cannt be cmpared t ne anther because f the timeframe issue Shadw prices In rder t use cst-benefit analysis, all csts and benefits must be mnetized. Hwever, as Karly states, a significant challenge in applying the cst-benefit apprach is t have apprpriate shadw prices r dllar values t attach t each f the shrt- and lng-term utcmes that a scial prgram may affect. 21 In sme cases, shadw prices are easy t btain, while in ther cases, their derivatin may be mre cmplex and subject t debate amng experts. Fr example, there is n market price fr the intangible cst f crime t a victim f crime. In practice, individual evaluatrs and researchers select their wn shadw prices s there is little cnsistency and therefre the results are nt easily cmparable Interdependencies Interdependencies refers t the idea that the utcmes f ne r a series f interventins are dependent n ther interventins. In rder t see change in X yu wuld als need t see change in Y. Fr example, the utcme f imprved high schl graduatin rates thrugh a peer tutring prgram is dependent in part n the level f functin f the family in which the student resides. This issue f interdependency is especially imprtant when the timeframe fr the prjected benefits f the interventin is further ut int the future. In early childhd interventins, much f the child s future achievements/benefits are dependent n what happens with the rest f the child s life circumstances and experiences. It is fictin t nt accunt fr the interdependencies in calculating the future benefits, yet very few appraches d accunt fr them. 18 Karly, Lynn A., M. Rebecca Kilburn, James H. Bigelw, Jnathan P. Caulkins, Jill S. Cannn. Chapter Tw: Overview f Cst and Outcme Analysis. RAND: Assessing Csts and Benefits f Early Childhd Interventin Prgrams: Overview and Applicatins t the Starting Early, Starting Smart Prgram. 19 Ibid, ftnte, p Karly, Lynn A. Valuing Benefits in Benefit-Cst Studies f Scial Prgrams. RAND, 2008, p Ibid. p. 3. Melinda T. Tuan 16

17 Only ne apprach, the imprved SROI Framewrk, tries t address the issue f interdependencies 22 in its methdlgy. The methdlgy incrprates interviews with varius stakehlders invlved with a single interventin (e.g. funders, the gvernment and ther nnprfits addressing the same r related issue, etc.) t understand hw their effrts are related t that interventin. The methdlgy then instructs the practitiner t assign a certain share f the prjected scial return n investment t the interventin based n that feedback. But this methd f estimating interdependencies is just that an estimate. Ultimately, determining what prprtin f an bserved change is due t the activities f a single rganizatin is methdlgically challenging given the cmplexity f change and the difficulty f determining what wuld have happened any way Value judgments Classical cst-effectiveness analysis and cst-benefit analysis (and therefre any apprach that incrprates these methdlgies) d nt currently incrprate a cnsistent apprach t dealing with value judgments. Each study reflects the values f the researcher as t hw the csts and benefits are distributed amng stakehlders and hw the varius utcmes are valued. One value judgment has t d with the value f a life. Fr example, in using DALYs and QALYs in healthcare, shuld ne weight the value f a thirty-year ld s life saved r imprved mre than that f a 70-year ld? Anther value judgment is that f distributin. One can argue that an additinal dllar t a pr persn is wrth mre than an additinal dllar t a wealthy persn, but hw much mre? There are many ther types f value judgments which are embedded in each methdlgy (e.g. deciding whether x utcme is really a benefit t persn y) and these all influence the results f the analyses. 22 nef refers t interdependencies as share f utcme r attributin in its reprt Scial Return n Investment: Valuing What Matters. Findings and Evaluatin frm a Pilt Study Schlten, Peter, Jeremy Nichlls, Sara Olsen, Brett Galimidi. Scial Return n Investment: A Guide t SROI Analysis. Lenthe Publishers, Melinda T. Tuan 17

18 5. BIG PICTURE ISSUES All f these technical limitatins and issues pint t the bigger picture issues invlved with implementing integrated cst appraches t measuring and/r estimating scial value, namely: the incnsistent use f language the lack f cmmn measures in the scial sectr the lack f quality data n scial impacts, utcmes, utputs, and cst the lack f incentives fr transparency unintended cnsequences inadequate utilizatin the cst f measurement 5.1. Incnsistent Use f Language There is currently n standard lexicn fr the scial impact measurement field. This results in a situatin where a variety f terms are used, smetimes imprecisely, t refer t the methds in the general class f cst and utcme analyses, including benefit-cst analysis and cst-effectiveness, amng thers. 23 Acrss the eight rganizatins and appraches prfiled fr this paper, we fund a wide spectrum f uses and definitins fr the same wrds and fund different wrds being used t describe the same calculatin r result. This lack f cnsistency in the use f language is a manifestatin f the nascent stage f the field f scial prgram evaluatin in the scial sectr. It als presents a significant challenge t thse lking t cmpare and cntrast methdlgies and results between varius rganizatins and prgrams. In particular, the wrds utcme and impact were used by multiple rganizatins t describe utputs in their methdlgies and calculatins. Fr example, Acumen Fund writes abut its BACO Rati as a rati f cst per utcme r cst per scial impact, 24 but as Brian Trelstad, Chief Investment Officer fr Acumen explained, The BACO is based slidly n utputs we dn t have the resurces t prve utcmes s we fcus n the clearest set f utputs, and even then there is little reliable infrmatin n the utput side. 25 Several rganizatins used different wrds r phrases t describe the same cncept f the calculatin f a philanthrpy s r nnprfit s share f the results f an interventin. In Rbin Hd s case, Weinstein described their estimatin f the Rbin Hd Factr as an assessment f prprtinality. After all, Rbin Hd s impact is nt always prprtinal t ur grant. Fr example, there are prgrams t which we give relatively small amunts f mney (as a percentage f a grantee s 23 Karly, Lynn A., M. Rebecca Kilburn, James H. Bigelw, Jnathan P. Caulkins, Jill S. Cannn. Chapter Tw: Overview f Cst and Outcme Analysis. RAND: Assessing Csts and Benefits f Early Childhd Interventin Prgrams: Overview and Applicatins t the Starting Early, Starting Smart Prgram. p Acumen Fund. Acumen Fund Cncept Paper: The Best Available Charitable Optin (BACO). (Draft). 1/24/ Trelstad, Brian. Re: Acumen Fund Best Available Charitable Optins (BACO) and Prtfli Data Management System (PDMS) t Melinda Tuan, 30 April Melinda T. Tuan 18

19 ttal budget) but which wuld cllapse entirely if we withdrew ur mney. 26 In talking abut the same cncept, the Hewlett Fundatin described their estimatin f the philanthrpy s cntributin which is calculated by cmbining the percentage f an individual philanthrpic rganizatin s cntributin relative t the verall philanthrpic cntributin needed t achieve the utcme and hw essential the philanthrpy s investment is t achieving the utcme. 27 In a different applicatin, nef used share f utcme r attributin t describe an effrt t measure the interdependencies the prprtin f a prgram s success truly attributable t the rganizatin s interventin in cases where utcmes are influenced by ther rganizatins and factrs and especially where the stakehlders bjectives can nly be achieved thrugh the cmbined effrts f mre than ne rganizatin. 28 At first glance, hwever, it can appear that nef s share f utcme is referring t the same kind f calculatin as Hewlett s philanthrpy s cntributin and Rbin Hd s Rbin Hd Factr. These are just a few examples which highlight the challenges invlved with researching, replicating, r even discussing these varius appraches t integrating cst int measuring and/r estimating scial value in the absence f a cmmn lexicn and language Lack f Cmmn Measures in the Scial Sectr Very few cmmn measures are currently being used t evaluate scial impact in the scial sectr, whether within a prgram area r acrss prgram areas. As Karly nted in her study f 39 scial prgrams with prven effectiveness and rigrus evaluatins, The use f the cst-benefit framewrk t evaluate scial prgrams requires the ability t place a value n the utcmes affected by the prgram. Ideally, such values wuld be attached t all utcmes and applied in a cnsistent manner acrss prgrams s results can be cmpared. Our review highlights the diverse array f utcmes affected by these prgrams Even prgrams that have a cmmn bjective (e.g., early childhd interventin) d nt necessarily incrprate cmmn measures int the evaluatins. 29 Karly explained that withut cmmn measures, yu can t make the argument that yu shuld invest in prgram x vs. prgram y because the utcmes are different. It s really a prblem f apples t ranges. 30 Even the very best methdlgy cannt cmpensate fr the lack f cmmn measures, as each interventin is measuring its results differently. Of the eight methdlgies we prfiled, nly ne rganizatin is cllecting cmmn measures acrss its entire prtfli f investments: REDF. Nt cincidentally, REDF has always had a prtfli made up f less than 20 rganizatins in the same grantmaking area: supprted emplyment thrugh scial enterprises. Karly argues 26 Weinstein, Michael. Re: Rbin Hd Fundatin Benefit-Cst Rati and Integrated Cst Appraches t Measuring Scial Impact t Melinda Tuan, 23 April Redstne Strategy Grup, LLC. Making Every Dllar Cunt: Hw Expected Return Can Transfrm Philanthrpy. April 10, The New Ecnmics Fundatin. Measuring Real Value: A DIY Guide t Scial Return n Investment. p Karly, Lynn A. Valuing Benefits in Benefit-Cst Studies f Scial Prgrams. RAND, p Karly, Lynn A. Re: Cst-Benefit Studies f Scial Prgrams t Melinda Tuan, 20 and 21 May Melinda T. Tuan 19

20 fr a set f guidelines that researchers wuld fllw t build a mre cnsistent cstbenefit research methdlgy, including the establishment f cmmn measures by prgram area. The health field is the ne area which stands ut in its use f cmmn measures, namely DALYs and QALYs, which then allws fr cmparisn f csteffectiveness ratis f all health interventins. Mst peple agree that the lack f cmmn measures within prgram areas limits the ability t cmpare results acrss prgram areas. Fr example, Cst effectiveness in the field f educatin is far behind what they ve dne in health. If we were t define ne magic rati r measure in a unifrm way acrss bth educatin and health we d have t water dwn all these great things we knw in health t accmmdate the immaturity in educatin evaluatin 31 explained Hilary Rhdes, Research Fellw, U.S. Educatin at CHIP. And, despite the maturity f measuring results in the health field, many f the technical issues discussed earlier present challenges and limitatins t hw CEA is being used in the health field. Fr example, there are cntinued debates abut which csts shuld be included and varying pinins n which f the differing value judgments (e.g. age weighting, distributin) shuld be embedded in varius analyses. 32 Susan Stut, recently retired Manager f the Wrld Bank s Results Secretariat, summed it up this way: cmparing results acrss prgram areas is pssible nly if we are willing t d the analytics t translate any result (utput r utcme) int a dllar (r yuan r eur) value. While DALYs mve the health field significantly frward t getting everyne t cnsider cst effectiveness they d nt extend t ther fields very well, which makes the really interesting cmparative judgments especially difficult Lack f Quality Data n Impacts, Outcmes, Outputs, and Csts Anther significant issue in measuring and/r estimating scial value, beynd the lack f cmmn measures, is the lack f quality data fr these measures verall. Karly cmmented n this issue in her bk chapter n hw t cnduct a cstbenefit analysis: The analyst must use creativity and infrmed guesswrk rarely will there be enugh data f high enugh quality that all entries (f csts and benefits) can be estimated with high cnfidence. Large blcks f entries may need t be based n educated guesswrk if they are nt t be left entirely blank. Of curse this affects the reliability f the analysis, but in ur view, it shuld nt be taken as an excuse t abandn analysis altgether. 34 In cnversatins with the designers f the eight appraches prfiled in this paper, the issue f the pr quality r ttal lack f data fr impacts, utcmes, utputs, and 31 Rsqueta, Kat, Hilary Rhdes, and Kathleen Nnan. Re: Center fr High Impact Philanthrpy Cst Per Impact Measures and ther Integrated Cst Appraches t Measuring Scial Impact t Melinda Tuan, 8 May Brck, Dan W. and Daniel Wikler. Ethical Issues in Resurce Allcatin, Research, and New Prduct Develpment Disease Cntrl Pririties in Develping Cuntries (2nd Editin), ed., New Yrk: Oxfrd University Press. DOI: / /Chpt Stut, Susan. Re: Perspectives n Cst-Benefit Analysis and Cst-Effectiveness Analysis in Glbal Health cmmunicatin t Philip Setel, April Karly, Lynn A., M. Rebecca Kilburn, James H. Bigelw, Jnathan P. Caulkins, Jill S. Cannn. Chapter Tw: Overview f Cst and Outcme Analysis. RAND: Assessing Csts and Benefits f Early Childhd Interventin Prgrams: Overview and Applicatins t the Starting Early, Starting Smart Prgram. p. 15. Melinda T. Tuan 20

21 cst and the implicatins fr their methdlgies arse multiple times. Kat Rsqueta, Executive Directr f CHIP described the prblem as fllws: The framewrk is sphisticated but the level f infrmatin t put int the framewrk is nt. When yu see hw there are layers and layers f garbage in, yu re ging t get garbage ut, n matter what the calculatin is. And then yu have t questin the usefulness f that calculatin. 35 CHIP s difficulties in btaining reliable cst infrmatin fr its Cst per Impact calculatins are reflected in Acumen Fund s challenges in btaining reliable infrmatin n scial utputs (nt t mentin utcmes) fr its prjected BACO Ratis. nef s Jeremy Nichlls lamented the lack f scial impact data saying smetimes the data just isn t there fr cntrl grups r cmparisn grups. Yu d think there wuld be sme system-wide result r benchmark fr results but there isn t. Until the quality f available scial impact, utcme, utput, and cst data imprves significantly, it will cntinue t be very difficult t measure scial value creatin with any degree f fidelity Lack f Incentives fr Transparency Even if the sectr was able t prduce quality data n scial utputs, utcmes, impact and cst, the questin remains whether there are any incentives fr philanthrpic and nnprfit rganizatins t share this infrmatin in a transparent fashin. As Trelstad nted, There is a fear f failure in the scial sectr. There may be a difference between hw endwed institutins versus thse wh have t g ut and raise it view this; but it s nt clear what incentives we have t shw ur lsers t ur dnrs. 36 Trelstad added that if the scial sectr is able t generate high quality data t allw analyses and cmparisns f rganizatins r prgrams based n their csteffectiveness there will be clear winners and lsers based n these analyses. Withut prper incentives fr rganizatins t be transparent abut their data, whether gd r bad, the prer results will likely be buried and nly the gd results shwcased t the detriment f the scial sectr as a whle Unintended Cnsequences A ppular adage states: Yu get what yu measure. Hwever, the challenge is: what yu measure may nt be what yu intended. It is imprtant t be aware f r try t predict the ptential unintended cnsequences f any effrt t integrate cst int measuring and/r estimating scial value. The very nature f unintended cnsequences is that they are ften unexpected which means it is difficult t plan fr them in advance. The prfiled rganizatins experienced a cuple cnsequences that that are wrth mentining fr future reference. One ptential unintended cnsequence is that nnprfits may try t game whatever scial value measurement system the funding rganizatin develps. In at least ne f the prfiled appraches, grantees f the funding rganizatin have been knwn t cherry-pick the prgrams they present t the fundatin fr funding. 35 Rsqueta, Kat, Hilary Rhdes, and Kathleen Nnan. Re: Center fr High Impact Philanthrpy Cst Per Impact Measures and ther Integrated Cst Appraches t Measuring Scial Impact t Melinda Tuan, 8 May Trelstad, Brian. Measuring and/r Estimating Scial Value Creatin Meeting, Bill & Melinda Gates Fundatin, Seattle, Octber 20-21, Melinda T. Tuan 21

22 Knwing the methds by which the funder calculates its scial value ratis, the nnprfits pick the prgrams that address the easiest-t-serve ppulatins which are mre likely t generate higher scres and therefre secure cntinued funding. Anther unintended cnsequence is that quantitative metrics will becme the sle fcus fr measuring scial value creatin, t the exclusin f qualitative analyses. Jed Emersn cautined, The metrics must be understd in the cntext f the narrative. It needs t be abut judgment, analysis and perspective, nt just the data. Emersn cmpared measuring scial value creatin t the light spectrum: The metrics are the visible light; the qualitative analyses are the gamma rays and ultravilet rays which are nt visible t the naked eye. N ne wuld argue that gamma and ultravilent rays d nt exist r are nt relevant, just because yu can t see them as easily. 37 Similarly, the scial sectr shuld nt lse perspective n the spectrum f measures fr value creatin, including bth quantitative and qualitative results. A third unintended cnsequence is the reality that any measurement effrt is an interventin in and f itself bth within a fundatin and with its grantees. Whatever a fundatin decides t emphasize in terms f hw t measure its scial impact will change the way fundatin staff think and act regarding the selectin f prgrams and rganizatins. Fundatin grantees that are evaluated thrugh a new measurement framewrk will als change in respnse, smetimes fr the better, smetimes fr the wrse as mentined earlier Inadequate Utilizatin In ur review f these appraches t integrating cst int measuring and/r estimating scial value creatin, it is clear that great effrt has been put int the develpment f each methdlgy. What is nt clear is whether these methdlgies are being used as riginally intended. It appears that in at least several cases, there is a great distance between the thery and practice. Several f the utilizatin issues are related t the technical and big picture issues discussed earlier. Cst-benefit analysis verall is intended t be used t cmpare ne prgram versus anther prgram. Hwever, due t the lack f cmmn, quality data, such cmparisns are inapprpriate. CHIP is encuntering similar challenges in develping its Cst per Impact methdlgy because f a lack f quality cst data. Additinally, it remains t be seen whether philanthrpists respnd well t CHIP s calculatins f Cst per Impact as their methdlgy is still being develped and tested. In ther cases, the rganizatin s methdlgy is described as essential t infrming decisins, yet in reality it appears that the methdlgy plays primarily a prmtinal rle fr the rganizatin. The results demnstrate t funders and bards f directrs that elegant analyses are being cnducted but the results are nt necessarily being used t infrm nging practice. Fr example, many f the rganizatins use methdlgies which invlve making multiple assumptins in rder t prject the future benefits f a particular investment decisin. Very few f the rganizatins have re-assessed thse 37 Emersn, Jed. Measuring and/r Estimating Scial Value Creatin Meeting, Bill & Melinda Gates Fundatin, Seattle, Octber 20-21, Melinda T. Tuan 22

23 assumptins n an annual basis due t the relative newness f the methdlgies, limited time and resurces, and in sme cases, lack f interest. This means that the assumptins in these methdlgies are nt being tested n an nging basis. And if the assumptins aren t being tested r adjusted, they will have little influence in infrming the develpment f mre accurate assumptins and calculatins. It is irnic that the utcmes f these methdlgies are nt ften being used t infrm the nging practice f using the same methdlgies. This may then lead t pr investment chices in the future. 5.7 Cst f Measurement Measuring data scial utputs, utcmes, impacts, and csts requires a lt f resurces. Cllecting and analyzing data can be very expensive and this expense is ften brne by the grantees f fundatins that require such data. Typically, nnprfits have limited time and mney t pursue activities utside f their missinbased prgramming. Additinally, mst nnprfits d nt have the administrative depth r expertise t track scial utcme and cst data. Ideally, funders will include the cst f data cllectin in their grants t funded nnprfit rganizatins. Hwever, even thugh in several cases, the funder tk n the majrity f the financial burden t implement the scial value measurement methdlgy (e.g. hiring third-party cnsultants t track the data, prviding grants fr infrmatin systems infrastructure), the prcess still required a significant investment f time frm the funded nnprfits. In a cuple cases, the funder underestimated the csts f measurement t bth the fundatin and grantees, as actual csts far exceeded riginal prjectins fr the evaluatin effrts. There is a cst/benefit t implementing any integrated cst methd fr measuring and/r estimating scial value. In REDF s case, the bard and senir management cncluded that their time and resurces, and that f their prtfli members, were better invested in tracking the individual scial utcmes f the prtfli enterprises than cntinuing t calculate returns t sciety as a whle thrugh its SROI framewrk. Frm inceptin, the intentin f REDF s SROI was never t cmpare the individual investments but rather argue fr the merits f investing in the prtfli f scial enterprise rganizatins and the field f scial enterprise as a whle. Hwever, when SROI was presented in its final frm t REDF s primary funder, Gerge R. Rberts, he asked whether the SROI results had changed the management team s investment decisins fr the prtfli. When the team respnded that it didn t change any f their decisins, and as the riginal intent was nt t use SROI t decide upn specific investments, Rberts suggested they discntinue calculating SROI metrics and instead fcus n cllecting and analyzing data that wuld infrm their nging investment decisins Emersn, Jed. Re: REDF s SROI t Melinda Tuan, 4 December Melinda T. Tuan 23

24 6. SUMMARY OF ISSUES AND IMPLICATIONS FOR THE SOCIAL SECTOR Five summary pints regarding the eight integrated cst appraches t measuring and/r estimating scial value creatin are wrth recunting: 1) Integrated cst appraches t measuring and/r estimating scial value are still in the nascent stages f develpment due t the lack f maturity in the field f scial prgram evaluatin. 2) The eight appraches prfiled represent a variety f philsphical purpses fr blending csts and scial utputs, utcmes, r impacts: internal decisinmaking crss prtflis, internal decisin-making within prtflis, and general prmtin and field building. They als serve varying practical purpses: making prspective investment decisins, infrming nging practice, and retrspectively evaluating philanthrpic investment decisins. 3) There is n perfect r precise slutin. Each methd has its strengths and weaknesses, and n single methd has been widely adpted thrughut the scial sectr. 4) There are many unreslved technical and big picture issues embedded in the methdlgies which determine the distance between the thery and the practice and affect verall utilizatin. 5) The lack f a cmmn language, cmmn measures, quality data, and incentives fr transparency represent key limitatins fr the utilizatin f any effrts t integrate cst int measuring and/r estimating scial value. In mving frward, it is essential fr the scial sectr t be very clear abut the purpse and benefit f creating and implementing an integrated cst apprach t measuring scial value; and the implicatins f pursuing any such apprach. In particular, it is imprtant t be mindful f the fllwing dangers: 1) The lure f false precisin: In reviewing all the detailed and smetimes quite cmplicated methdlgies, it can be easy t be cnvinced f the certainty f the results f these seemingly precise calculatins. 2) The desire fr a silver bullet: It is tempting t fcus n a single numeric t indicate whether an investment is successful r nt. Hwever, scial value metrics shuld be interpreted in their greater cntext in rder t make the best investment decisins. 3) The risk f cherry-picking: Cst-benefit metrics may verwhelmingly indicate that ne interventin shuld be favred ver anther. Yet smetimes the prblems that are the mst cst-effective t slve d nt end up fcusing n the neediest r hardest t serve ppulatins. It is crucial t nte that any high-fidelity apprach the scial sectr develps t integrate csts int measuring scial value will be limited by and directly affected by the sectr s ability t prduce high quality data. Ultimately, the sectr s largest effrts will nt be abut chsing the right mdel r methd. Rather, the mst significant effrt will invlve getting the right data t make whichever mdel r methdlgy a fundatin r nnprfit rganizatin chses useful. Withut high quality data, any practitiner s results will be based n ne assumptin after anther Melinda T. Tuan 24

25 r layers and layers f garbage. If the scial sectr is interested in creating mre precise, meaningful appraches t measuring and/r estimating scial value, fundatins will need t invest in increasing the quality f the scial and cst data infrastructure acrss the varius prgram areas represented in the scial sectr. Lastly, it is imprtant t emphasize that any data, high quality r nt; and any mdel fr analyzing data, high fidelity r nt, are subject t interpretatin. The same data can be interpreted by different peple and rganizatins t reach diametrically ppsed cnclusins. The true value f high quality data and analyses f any integrated cst apprach t measuring and/r estimating scial value creatin will be t stimulate high quality cnversatins abut the implicatins. In clsing, there are tw imprtant questins fr the scial sectr t cnsider in light f the lessns learned and insights frm the prfiled prmising practices fr integrating cst int measuring and/r estimating scial value: 1) What is the primary purpse fr the scial sectr t pursue an integrated cst apprach t measuring and/r estimating scial value? and, 2) What will the sectr d with the resulting infrmatin? Melinda T. Tuan 25

26 7. AUTHORSHIP AND RESEARCH CREDITS This reprt was written and researched by Melinda T. Tuan. Melinda is an independent cnsultant wh wrks with the senir leadership f philanthrpic rganizatins t research and create cntent regarding strategies fr thughtful and effective philanthrpy. In additin t her current wrk with the Bill & Melinda Gates Fundatin, Melinda recently cmpleted fur years as a Senir Fellw with Rckefeller Philanthrpy Advisrs and tw years as a Special Advisr t Bridgestar/The Bridgespan Grup. Other recent clients include Grantmakers fr Effective Organizatins and the Eagles Yuth Partnership, the philanthrpic arm f the Philadelphia Eagles Ftball Team. Previusly, Melinda c-funded and managed REDF (frmerly The Rberts Enterprise Develpment Fund) with Jed Emersn and Gerge Rberts f KKR. REDF is a scial venture capital fund that wrks with a prtfli f nnprfit rganizatins emplying frmerly hmeless and lw-incme individuals in market-based business ventures. While at REDF, Melinda invested in a prtfli f fifteen nnprfit rganizatins running ver thirty different businesses emplying ver 2,000 frmerly hmeless and lw-incme individuals ver a seven-year perid. In additin, she crdinated the design and develpment prcess fr REDF s scial return n investment (SROI) framewrk which was architected by Jed Emersn, and managed the develpment f REDF s nging assessment f scial impacts (OASIS). Prir t REDF, Melinda was a manager at a natinal healthcare nnprfit and a management cnsultant specializing in grwth strategies fr Frtune 500 cmpanies. Melinda has vlunteered with numerus cmmunity-based rganizatins in Hnlulu, Bstn, and the San Francisc Bay Area that serve hmeless and very lw-incme ppulatins. Additinally, she c-funded Bstn Cares, a nnprfit vlunteer service rganizatin, and was invlved in the start-up f a scial-missin driven cmpany called Dayspring Technlgies in San Francisc. Melinda is recgnized natinally fr her wrk in high engagement philanthrpy, fundatin effectiveness, evaluatin, nnprfit capacity-building, and scial enterprise. She has lectured at leading business schls in the cuntry including Stanfrd and Whartn, and published articles, business schl cases, and a bk chapter entitled Cultivating a Culture f Measurement in Funding Effectiveness by Grantmakers fr Effective Organizatins. Melinda currently serves n the Bard f Managers fr Evergreen Ldge, a scial-purpse destinatin resrt lcated just utside Ysemite Natinal Park, and the Advisry Cuncil fr REDF. Melinda graduated frm Harvard University magna cum laude with an AB in Scial Studies fcusing n urban pverty and hmelessness and she hlds an MBA and certificate in nnprfit management frm the Stanfrd Graduate Schl f Business. She resides in Narberth, Pennsylvania with her husband and three children. Melinda T. Tuan 26

27 APPENDIX A: GLOSSARY OF TERMS Assumptins: What yu have t believe t be true in rder t have cnfidence in X a belief. Baseline: A state f the wrld withut the prgram that can be cmpared t the wrld with the prgram in place. 39 Benefit-Cst Rati: One f tw cmmn ways t cmpare the benefits and csts f an interventin. Dividing the mnetized benefits by the mnetized csts yields a benefit-cst rati. A rati f greater than ne means the benefit is greater than the cst and a rati f less than ne means the cst is greater than the benefit. Causal Effects/Causality: The full range f tangible and intangible utcmes that may be affected by the prgram. These utcmes are ideally captured in a welldesigned and well-implemented randmized experimental design evaluatin, where members f the target ppulatin fr the prgram are randmly assigned t participate r nt participate in the prgram. The difference in utcmes between the tw grups can be calculated as the impact caused by the prgram. Cmmn Measures: Standard measures f impact (utcmes) that can be used acrss a variety f prgrams in a field f study (e.g. IQ scres fr children within the field f educatin). Cst Analysis: Generates a measure f the prgram cst based n a cmprehensive measurement f the ecnmic value f the resurces required fr prgram implementatin. 40 Cst-Benefit Analysis (CBA): Takes the perspective f sciety as a whle and cnsiders the csts and dllar-valued utcmes aggregated acrss all stakehlders (gvernment sectr r individuals as taxpayers, prgram participants r private individuals, the rest f sciety). The utput frm cst-benefit analysis can be measures f net benefits (benefits csts), the rati f benefits t cst (benefit-cst ratis), r the internal rate f return (the rate f grwth a prject is expected t generate). By requiring cmprehensive measurement f csts and prgram impacts, and the ability t place a dllar value n prgram impacts acrss stakehlders, CBA is the mst demanding f the cst and utcme analysis appraches. At the same time, it is als the mst cmprehensive in prviding a full accunting f the net benefits t sciety as a whle, as well as varius stakehlders. 41 Using educatin as an example, ne wuld calculate the mnetary value f having an educated child (e.g. measured in terms f human capital r increased ecnmic prductivity) minus the cst f educating the child. The units fr CBA are simply 39 Karly, Lynn A., M. Rebecca Kilburn, James H. Bigelw, Jnathan P. Caulkins, Jill S. Cannn. Chapter Tw: Overview f Cst and Outcme Analysis. RAND: Assessing Csts and Benefits f Early Childhd Interventin Prgrams: Overview and Applicatins t the Starting Early, Starting Smart Prgram. p Karly, Lynn A. Valuing Benefits in Benefit-Cst Studies f Scial Prgrams. RAND, 2008, p Ibid. Melinda T. Tuan 27

28 dllars (r eurs, r yen), nt a rati. In an ideal wrld, CBA allws ne t cmpare apples t ranges in that everything is mnetized. Cst-Effectiveness Analysis (CEA): The calculatin f a rati f cst t a nnmnetary benefit. The fcus may be n ne dmain f impact (e.g. crime, student achievement) r multiple areas f impact. Hwever, measures f cst-effectiveness can nly accunt fr ne area f prgram impact at a time. Since prgram impacts are measured in natural units (e.g. life year saved, child graduating frm high schl), unless thse units are cmmn acrss all areas f impact, it is nt pssible t aggregate acrss them. 42 The units fr CEA are ratis (cst per smething). Fr example: cst per quality-adjusted life year (QALY) r disability-adjusted life year (DALY) cst per persn cured f malaria cst per child educated fr ne additinal year cst per autmbile accident avided It is cmmn t invert the rati, calculating the cst per unit f benefit purchased. Fr instance, health care prgrams are ften evaluated in terms f the cst per QALY saved. In thse cases, smaller numbers indicate mre efficient prgrams. The csteffectiveness rati fr a single prgram is ften difficult t interpret withut knwing the cntext, but if ne calculates the cst-effectiveness rati fr each available interventin, the ne with the highest rati is the preferred place t invest the next dllars. (If the ratis are cmputed in terms f cst per unit benefit, nt benefit per unit cst, then the interventin with the smallest rati wuld be preferred). 43 Cst Savings Analysis: A term smetimes used t refer t a cst-benefit analysis dne frm the perspective f the gvernment generally r a particular gvernment agency. It cmpares nly the csts t gvernment fr prgram implementatin and the savings (r csts) t gvernment generated frm a prgram and its assciated prgram impacts. Cst savings analysis is used when asking questins such as whether the benefits f a prgram t gvernment pay back the csts taxpayers invested in the prgram. 44 Cst savings analysis values all prgram impacts in dllars. The utput frm cst-savings can be measures f net savings (savings cst), the rati f savings (savings-cst ratis), r the internal rate f return. 45 While this term is used in the vernacular t mean many things, Cst Savings Analysis is a technical term used by ecnmists t evaluate the benefits f public funding streams. 42 Karly, Lynn A. Valuing Benefits in Benefit-Cst Studies f Scial Prgrams. RAND, 2008, p Ibid. 44 Karly, Lynn A., M. Rebecca Kilburn, James H. Bigelw, Jnathan P. Caulkins, Jill S. Cannn. Chapter Tw: Overview f Cst and Outcme Analysis. RAND: Assessing Csts and Benefits f Early Childhd Interventin Prgrams: Overview and Applicatins t the Starting Early, Starting Smart Prgram. p Karly, Lynn A. Valuing Benefits in Benefit-Cst Studies f Scial Prgrams. RAND, 2008, p Melinda T. Tuan 28

29 Table 1.1 Types f Cst and Outcme Analysis and Assciated Infrmatin Requirements 46 Type f Analysis Outcme f Analysis Infrmatin Requirement Cst Cst f Prgram Cmprehensive measure f prgram csts Cst-effectiveness Measure f cst per unit change in a specific Cmprehensive measure f prgram csts utcme, value fr ne impact at a time Measures f prgram impacts in natural units Cst-savings Cst-benefit Measure f net savings t gvernment, inclusive f all impacts Measure f rati f gvernment savings t csts Measure f internal rate f return t gvernment Measure f net benefit t sciety, inclusive f all impacts 47 Measure f rati f benefits t csts Measure f rate f return t sciety Cmprehensive measure f prgram csts, specific t gvernment sectr Measures f prgram impacts at each pint in time in natural units Shadw prices t value all utcmes in dllars, specific t gvernment sectr Cmprehensive measure f prgram csts at each pint in time, in aggregate and specific t varius stakehlders Measures f prgram impacts at each pint in time in natural units Shadw prices t value all utcmes in dllars, in aggregate and specific t varius stakehlders Cst-Utility Analyses: A type f cst-effectiveness analysis where utcmes include a quality f life cmpnent (e.g. QALYS, DALYS). Disability-Adjusted Life Year (DALY): The DALY relies n an acceptance that the mst apprpriate measure f the effects f chrnic illness is time, bth time lst due t premature death and time spent disabled by disease. One DALY, therefre, is equal t ne year f healthy life lst. 48 When calculated, the DALY is the number f years f life lst due t premature death (cmpared t a standard life expectancy) plus the years f life lived in a state f less than full health. 49 The principal difference between QALYs and DALYs is that QALY weightings are derived by asking patients t rate their health status whereas in DALYs the weightings are derived by asking 46 Karly, Lynn A. Valuing Benefits in Benefit-Cst Studies f Scial Prgrams. RAND, 2008, p When csts and/r benefits accrue ver multiple time perids, the dllar streams are discunted t reflect the time value f mney. Thus, the relevant utcme is net present value savings r benefit. 48 Wikipedia Melinda T. Tuan 29

30 health experts r the general public t rate a whle series f health states (e.g. if ne lst a limb, became blind, was cnfined t a wheelchair.) 50 Discunting: The practice f weighing r valuing utcmes that ccur sner mre than utcmes that are delayed. It is bvius why this shuld be s with mney. One wuld rather have $1,000 tday than $1,000 next year, because if a persn had $1,000 tday he r she culd invest it and have mre than $1,000 next year. The same lgic f discunting r applying time preferences can be applied t nnmnetary utcmes, and at the same rate. 51 Discunt Rate: The discunt rate is a financial metric that may be used t determine the present value f future payments r expenditures. 52 Expected Value: A term used by mathematicians t represent the average amunt ne "expects" as the utcme f the randm trial when identical dds are repeated many times. The value itself may nt be expected in the general sense the "expected value" itself may be unlikely r even impssible. 53 Fr example, peple buying a lttery ticket that has a 1/10,000 chance f paying $10,000 can expect t get zer since that is verwhelmingly the likely utcme. They can be certain they wn't get $1. But the expected value f their winnings is $1. 54 Nte: this definitin f expected value is different frm hw Hewlett Fundatin describes its methdlgy which is named Expected Value. Impacts: The lng-term sustainable and smetimes attributable change due t a specific interventin r set f interventins. Interdependence/Interdependencies: The idea that the utcmes f ne r a series f interventins are dependent n ther interventins. Fr example, the utcme f imprved high schl graduatin rates thrugh a peer tutring prgram is dependent in part n the level f functin f the family in which the student resides. In rder t see change in X yu wuld als need t see change in Y. Internal Rate f Return: IRR is a strictly (theretical) mathematical frmula and is ne f the many ways return n investment (ROI) can be measured. One can think f IRR as the rate f grwth a prject is expected t generate. In the fllwing equatin, ne wuld calculate the IRR by slving fr r where the net present value (NPV) f the investment equals 0 and I is the prjected cash flw in year 0, 1, 2, etc Karly, Lynn A., M. Rebecca Kilburn, James H. Bigelw, Jnathan P. Caulkins, Jill S. Cannn. Chapter Tw: Overview f Cst and Outcme Analysis. RAND: Assessing Csts and Benefits f Early Childhd Interventin Prgrams: Overview and Applicatins t the Starting Early, Starting Smart Prgram. p Wikipedia. 53 Ibid Melinda T. Tuan 30

31 IRR is ften used in capital budgeting and primarily shws the value anther investment wuld need t generate in rder t be equivalent t the cash flws f the investments being cnsidered. IRR illustrates verall returns in clear percentage terms and is great fr cmparing prject returns head t head. Generally speaking, the higher a prject's internal rate f return, the mre desirable it is t undertake the prject. As such, IRR can be used t rank several prspective prjects a firm is cnsidering. Assuming all ther factrs are equal amng the varius prjects, the prject with the highest IRR wuld prbably be cnsidered the best and undertaken first. 55 Hwever, IRR des nt indicate the cmparative level f investment required upfrnt r the verall dllar f returns. 56 Inputs: The resurces used t run the prgram: the mney, peple, facilities, and equipment. Natural Unit: Natural units are utcmes measured in nn-mnetary terms. They are typically used in cst-effectiveness analysis as the denminatr f the csteffectiveness rati (cst per natural unit x). Examples f natural units include life year saved and child graduating frm high schl. Natural units are nt necessarily als cmmn measures. Net Present Value: One f tw cmmn ways t cmpare benefits and csts by lking at their difference. Subtracting mnetized csts frm mnetized benefits yields the net value. Because discunting is ften invlved, this is mst ften called the net present value, r NPV. Net Value: (see Net Present Value ) Outcmes: The changes that ccur ver time fllwing an interventin r set f interventins. Outcmes can be measured at a variety f levels: individual, rganizatinal, cmmunity, system, funding stream, etc... Outcmes may be direct r indirect. Direct utcmes fllw frm the utputs (e.g. getting a jb) and indirect utcmes fllw frm the direct utcmes (e.g. increase in incme due t the jb gained). Outputs: The direct and tangible prducts frm the activity (e.g. the number f peple trained). Payback Perid: The length f time a prgram must remain in peratin t recup the initial investment. 57 Present Value: The value tday f an amunt f mney in the future. 58 The idea is that given a discunt rate (e.g. 4%); ne shuld feel the same abut receiving $57,700 tday and receiving $20,000 at the end f each f the next three years. In 55 Investpedia. 56 Pisell, Tm. What s the Difference Between NPV and IRR? [Available Online] 57 Karly, Lynn A., M. Rebecca Kilburn, James H. Bigelw, Jnathan P. Caulkins, Jill S. Cannn. Assessing Csts and Benefits f Early Childhd Interventin Prgrams: Overview and Applicatins t the Starting Early, Starting Smart Prgram. RAND, [Available Online] p Hendersn, David R. The Cncise Encyclpedia f Ecnmics. [Available Online] Melinda T. Tuan 31

32 terms f nn-mnetary utcmes, ne culd discunt 100 emergency rm visits per year fr the next three years by the same rate t get a present value f 289 visits. 59 Quality-Adjusted Life Year (QALY): A single measure f health utcme that simultaneusly captures gains frm reduced mrbidity (quality f life gains) and reduced mrtality (quantity f life gains). 60 QALYs are calculated by multiplying the number f years f life that wuld be added by the interventin by the imprvement in quality f life frm that interventin (measured n a scale between 0 and 1 where 1 is a state f full health and 0 is the wrst pssible health state). The principal difference between QALYs and DALYs is that QALY weightings are derived frm asking patients t rate their health status whereas in DALYs the weightings are derived by asking health experts r the general public t rate a whle series f health states (e.g. if ne lst a limb, became blind, was cnfined t a wheelchair.) 61 Quasi-Experimental Designs: Evaluatin research that includes a cmparisn r cntrl grup chsen n the basis f matched characteristics but nt randm assignment. 62 Quasi-experimental design evaluatins are cnsidered t deliver smewhat less certainty than results frm randmized experimental design evaluatins, but mre certainty than pre-pst evaluatins. This methd is used when finding randmly assigned grups is nt pssible r apprpriate. Randmized Experimental Designs: Evaluatin research cnducted whereby the cntrl and treatment grups are as similar as pssible except fr participatin in the prgram. In experimental evaluatins, individuals are randmly assigned t the cntrl grup (i.e., the grup that receives n new prgram services r faces the status qu) r the treatment grup (i.e., the grup that receives the prgram services r faces the plicy alternative). Thus, any differences can be attributed t the impact f the prgram r plicy. 63 Return n Investment (ROI) and Rate f Return (ROR): In finance, rate f return (ROR) r return n investment (ROI), r smetimes just return, is the rati f mney gained r lst n an investment relative t the amunt f mney invested. ROI is usually given as a percent rather than decimal value. ROI is als knwn as rate f prfit. ROI des nt indicate hw lng an investment is held. Hwever, ROI is mst ften stated as a percentage in an annual r annualized rate f return, and it is mst ften stated fr a calendar r fiscal year. 64 ROI is used t cmpare returns n investments where the mney gained r lst r the mney invested is nt easily cmpared using mnetary values. Fr instance, a $1,000 investment that earns $50 in interest bviusly generates mre cash than a 59 Karly, Lynn A., M. Rebecca Kilburn, James H. Bigelw, Jnathan P. Caulkins, Jill S. Cannn. Chapter Tw: Overview f Cst and Outcme Analysis. RAND: Assessing Csts and Benefits f Early Childhd Interventin Prgrams: Overview and Applicatins t the Starting Early, Starting Smart Prgram. p Ibid. 62 Karly, Lynn A., M. Rebecca Kilburn, James H. Bigelw, Jnathan P. Caulkins, Jill S. Cannn. Chapter Tw: Overview f Cst and Outcme Analysis. RAND: Assessing Csts and Benefits f Early Childhd Interventin Prgrams: Overview and Applicatins t the Starting Early, Starting Smart Prgram. pp Ibid. 64 Wikipedia. Melinda T. Tuan 32

33 $100 investment that earns $20 in interest, but the $100 investment earns a higher return n investment. $50/$1,000 = 5% ROI $20/$100 = 20% ROI Shadw Prices: Shadw prices are dllar values that are attached t each f the shrt and lng-term utcmes that a scial prgram may affect. Shadw prices are typically used in cst-benefit analyses. In sme cases, such ecnmic values may be readily btained, while in thers, their derivatin may be mre cmplex and subject t debate amng experts. 65 One example f a difficult t btain ecnmic value is the intangible cst f crime fr crime victims. There is n market price fr this intangible item; therefre a shadw price must be develped fr it fr use in a cstbenefit analysis. Scial Impact: (see Impact) Scial Return n Investment (SROI): A term ppularized by REDF in the late 1990s that nw has widespread use in bth the nnprfit and increasingly fr-prfit sectrs fr describing any number f appraches t estimating r calculating the scial utput r utcmes r impact f a prgram r enterprise. There is currently n standard definitin fr SROI althugh it is widely referenced in the wrk f nnprfits, philanthrpy, and scially respnsible businesses. 65 Karly, Lynn A. Valuing Benefits in Benefit-Cst Studies f Scial Prgrams. RAND, p. 3, p. ix. Melinda T. Tuan 33

34 APPENDIX B: LEADING PRACTITIONER AND EXPERT INTERVIEW LIST Persn Affiliatin Tpic Paul Brest, President William and Flra Hewlett Fundatin Hewlett Fundatin Expected Return; Evlutin and applicatin f SROI J. Gregry Dees, Prfessr Fuqua Schl f Business, Duke University Jed Emersn, Managing Directr fr Integrated Perfrmance Uhuru Capital Management Histry f develpment f financial markets, crllary t scial capital market Origins f the fr-prfit and scial capital markets; blended value investing Funding Directr, REDF REDF Scial Return n Investment (SROI) Lynn A. Karly, Ph.D., RAND Crpratin Cst-benefit analysis Senir Ecnmist Authr f MacArthur Cst-effectiveness analysis Fundatin funded study f cst/benefit analysis effrts in valuing scial prgrams Kieran McGrath Frmer Senir Prgram Utilizatin f Rbin Hd Officer, Rbin Hd Benefit-Cst Rati Fundatin Jeremy Nichlls, Fellw The New Ecnmics Evlutin and applicatin f Fundatin SROI Chief Executive, SROI UK Sara Olsen, Funding SVT Grup Evlutin and applicatin f Partner SROI, bubble chart display Kat Rsqueta, Executive Center fr High Impact Center fr High Impact Directr; Kathleen Nnan, Philanthrpy, Schl f Philanthrpy Cst per Impact Assciate Directr; Hilary J. Scial Plicy & methdlgy Rhdes, Ph.D., Research Practice, University f Fellw, U.S. Educatin; Pennsylvania Carl McLaughlin, MD, MPH, Research Directr, Glbal Public Health Peter Schlten, CEO Schlten & Franssen Evlutin f SROI in Eurpe Susan Stut, Recently Wrld Bank s Results Value and challenges f Retired Manager Secretariat cmparing results acrss sectrs, silver bulletism Brian Trelstad, Chief Acumen Fund Acumen Fund Best Available Investment Officer Charitable Optin (BACO) Michael M. Weinstein, Chief Prgram Officer Rbin Hd Fundatin Rbin Hd Fundatin Benefit/Cst Rati Melinda T. Tuan 34

35 APPENDIX C: MEASURING AND/OR ESTIMATING SOCIAL VALUE CREATION MEETING PARTICIPANTS OCTOBER 20 21, 2008 The Bill & Melinda Gates Fundatin wuld like t acknwledge and thank the fllwing leading practitiners and experts and internal fundatin participants fr their thughtful and stimulating participatin in the Scial Value Creatin Meeting. These participants individually and cllectively helped advance the Fundatin s thinking n pragmatic and rigrus ways t measure and/r estimate scial value creatin and have already helped advance the field thrugh their input t this paper. External Participant List Ivan Barkhrn Managing Directr Redstne Strategy Grup Paul Brest President William and Flra Hewlett Fundatin Chris DeCardy Vice President and Directr f Cmmunicatins The David and Lucile Packard Fundatin Jed Emersn Managing Directr fr Integrated Perfrmance Uhuru Capital Management, LLC Lynn A. Karly, PhD Senir Ecnmist RAND Crpratin Christina Leijnhufvud Managing Directr Investment Bank/Scial Sectr Finance JPMrgan Chase Brad Presner Metrics Manager Acumen Fund Kat Rsqueta Executive Directr Center fr High Impact Philanthrpy Susan Stut Retired Manager Wrld Bank Results Secretariat Brian Trelstad Chief Investment Officer Acumen Fund Michael Weinstein Chief Prgram Officer Rbin Hd Fundatin Melinda Tuan Cnsultant Bill & Melinda Gates Fundatin Carl A. McLaughlin, MD, MPH Research Directr, Glbal Public Health Center fr High Impact Philanthrpy Sara Olsen Funding Partner SVT Grup Melinda T. Tuan 35

36 APPENDIX C: MEASURING AND/OR ESTIMATING SOCIAL VALUE CREATION MEETING PARTICIPANTS OCTOBER 20 21, 2008 (cntinued) Fundatin Participant List Rashmir Balasubramaniam Prgram Officer, Water, Sanitatin & Hygiene Glbal Develpment Eleanr Bell Senir IPI Officer Impact Planning and Imprvement Crss Fundatin Jim Brmley Deputy Directr Financial Planning and Analysis Jana Carlisle Senir IPI Officer Impact Planning and Imprvement- United States Michael Deich Directr Plicy & Gvernmental Affairs Brittany Faulkner Senir Assistant Impact Planning and Imprvement Crss Fundatin Elvis Fraser Deputy Directr Impact Planning and Imprvement Crss Fundatin Alex Friedman Chief Financial Officer Gargee Ghsh Sr. Prgram Officer, Develpment Finance & Plicy Glbal Health Kendall Guthrie Interim Deputy Directr Impact Planning and Imprvement United States Kim Hamiltn Deputy Directr, Plicy & Advcacy Glbal Develpment Debrah Jacbs Deputy Directr, Glbal Libraries Glbal Develpment Elisa Mandell Special Plicy Officer Glbal Develpment Sara Meyer IPI Officer Impact Planning and Imprvement- United States Keith Olsn Directr Financial Planning & Analysis Casey Owens Special Assistant t CFO Dan Radcliffe Assciate Prgram Officer Glbal Develpment Abbie Raikes Senir IPI Officer Impact Planning and Imprvement Crss Fundatin Jeff Raikes Chief Executive Officer Melinda T. Tuan 36

37 Amy Ratcliffe Senir IPI Officer Impact Planning and Imprvement Glbal Health Jeffrey Ried Prgram Officer Glbal Develpment Philip Setel Deputy Directr Impact Planning and Imprvement Glbal Health Fay Twersky Directr Impact Planning and Imprvement Melinda T. Tuan 37

38 APPENDIX D: BIBLIOGRAPHY Acumen Fund. Acumen Fund Cncept Paper: The Best Available Charitable Optin (BACO). (Draft). 1/24/07. Acumen Fund. Develping a glbal prtfli data management system: Transparency and accuntability fr the field f private sectr develpment. (Internal cnfidential dcument). Acumen Fund. Investment Discipline. [Available nline] Acumen Fund. Prject Yaz Business Plan: A Scial Enterprise Impact Assessment Platfrm. (Internal cnfidential dcument). Acumen Fund. The Acumen Fund and Ggle Prtfli Data Management System (Internal cnfidential dcument). Aern-Thmas, David, Jeremy Nichlls, Sarah Frster, Andrea Westall. Scial Return n Investment: Valuing What Matters. Findings and Evaluatin frm a Pilt Study. New Ecnmics Fundatin [Available Online] Brest, Paul and Hal Harvey. Assessing Investment Opprtunities in Internatinal Develpment: The Acumen Fund s BACO Analysis. (Unpublished) Brest, Paul and Hal Harvey. Chapter 10: Impact n Sterids: Measuring the Scial Return n Yur Philanthrpic Investment. Mney Well Spent: A Strategic Guide t Smart Philanthrpy. Blmberg Press. Nvember Brest, Paul and Hal Harvey. New Ecnmics Fundatin. (Unpublished) Brest, Paul, Hal Harvey and Kelvin Lw. Calculated Impact. Stanfrd Scial Innvatin Review. Winter Brest, Paul. Re: Hewlett Fundatin s Expected Return Methdlgy t Melinda Tuan, 24 April Brck, Dan W. and Daniel Wikler. Ethical Issues in Resurce Allcatin, Research, and New Prduct Develpment Disease Cntrl Pririties in Develping Cuntries (2nd Editin), ed., New Yrk: Oxfrd University Press. DOI: / /Chpt-14. Center fr High Impact Philanthrpy, Schl f Scial Plicy & Practice, University f Pennsylvania. Children s Literacy Initiative Case Example Draft frm. May 5, Center fr High Impact Philanthrpy, Schl f Scial Plicy & Practice, University f Pennsylvania. What is High Impact Philanthrpy? [Available Online] Melinda T. Tuan 38

39 hrpy_initialcnceptpaperapril2007.pdf draft April 24, Center fr Impact Evaluatin. When Will We Ever Learn? Imprving Lives Thrugh Impact Evaluatin. May Clark, Cathy et al. Duble Bttm Line Prject Reprt: Assessing Scial Impact in Duble Bttm Line Ventures. Methds Catalg. [Available Online] February Enhanced Analytics Initiative. [Available Online] Emersn, Jed. Re: Integrated Cst Appraches t Measuring Scial Impact and REDF s SROI t Melinda Tuan, 1 May Emersn, Jed and Sheila Bnini. Blended Value Executive Summary. [Available Online] Octber 21, Fuchs, Victr R. Perspective: Mre Variatin In Use Of Care, Mre Flat-Of-The-Curve Medicine Health Affairs. 7 Octber Gair, Cynthia. A Reprt frm the Gd Ship SROI. The Rberts Fundatin [Available Online] REDF Glbal Scial Venture Cmpetitin. GSVC Scial Impact Assessment Guide 3 f 3: Mnetizing Scial Value / SROI. [Available Online] Hecht, Rbert and Raj Shah. Recent Trends and Innvatins in Develpment Assistance fr Health Disease Cntrl Pririties in Develping Cuntries (2nd Editin), ed., New Yrk: Oxfrd University Press. DOI: / /Chpt-13. Huggett, Jn. Business Planning: What it Is and Why it Matters. Pwer Pint Presentatin. The Bridgespan Grup. Page 7. Jamisn, Dean T. et al. Disease Cntrl Pririties in Develping Cuntries (2 nd Editin), ed. New Yrk: Oxfrd University Press. [Available Online] Karly, Lynn A. Re: Cst-Benefit Studies f Scial Prgrams t Melinda Tuan, 20 and 21 May Karly, Lynn A. and James H. Bigelw. The Ecnmics f Investing in Universal Preschl Educatin in Califrnia. RAND, [Available Online] Karly, Lynn A., M. Rebecca Kilburn, James H. Bigelw, Jnathan P. Caulkins, Jill S. Cannn. Chapter Tw: Overview f Cst and Outcme Analysis. RAND: Assessing Csts and Benefits f Early Childhd Interventin Prgrams: Overview and Applicatins t the Starting Early, Starting Smart Prgram. Melinda T. Tuan 39

40 [Available Online] p Karly, Lynn A. Valuing Benefits in Benefit-Cst Studies f Scial Prgrams. RAND, Kilburn, M. Rebecca and Lynn A. Karly. The Ecnmics f Early Childhd Plicy: What the Dismal Science Has t Say Abut Investing in Children. RAND [Available Online] Lamb, Geff. Financing fr Success: Outline f Wrk. Bill & Melinda Gates Fundatin internal mem. 4/27/07. Lamb, Geff. Fundatin prgrams and additinal surces f finance: Planning fr scale and sustainability. Bill & Melinda Gates Fundatin internal mem. 4/27/07. Levin, Henry M., Gene V. Glass, Gail R. Meister. Cst-Effectiveness f Cmputer- Assisted Instructin. Evaluatin Review, Vl. 11, N. 1, (1987). Levin, Henry M. Waiting fr Gdt: Cst-Effectiveness Analysis in Educatin. New Directins fr Evaluatin, n. 90. Jssey-Bass: Summer Lndn Business Schl. SROI Primer. [Available Online] Masn, Jhn B., David Sanders, Philip Musgrve, Sekirman, and Rae Gallway, "Cmmunity Health and Nutritin Prgrams." Disease Cntrl Pririties in Develping Cuntries (2nd Editin), ed., 1,053-1,074. New Yrk: Oxfrd University Press. DOI: / /Chpt-56. [Available Online] McGrath, Kieran. Re: Rbin Hd Fundatin Benefit-Cst Rati and Use by RHF Prgram Officers t Melinda Tuan, 7 May McLaughlin C., Nnan K., & Rsqueta K. Hw We Calculated Cst Per Impact in Glbal Health Prgrams Malaria. Bednets & Beynd: An Individual Philanthrpist s Guide t Addressing Malaria. Center fr High Impact Philanthrpy, Schl f Scial Plicy & Practice, University f Pennsylvania. Expected release Fall/Winter Measuring and/r Estimating Scial Value Creatin Meeting, Bill & Melinda Gates Fundatin, Seattle, Octber 20-21, Meltzer, David. "Ecnmic Appraches t Valuing Glbal Health Research." Disease Cntrl Pririties in Develping Cuntries (2nd Editin), ed., New Yrk: Oxfrd University Press. DOI: / /Chpt- 7. Musgrve, Philip and Julia Fx-Rushby. Cst-Effectiveness Analysis fr Pririty Setting Disease Cntrl Pririties in Develping Cuntries (2nd Melinda T. Tuan 40

41 Editin), ed., New Yrk: Oxfrd University Press. DOI: / /Chpt-15. Neuhff, Alex and Rbert Searle. Mre Bang fr the Buck. Stanfrd Scial Innvatin Review. Vlume 6, Number 2, Spring pp New Prgressive Calitin. PROI Framewrk. [Available Online] Nichlls, Jeremy. Re: New Ecnmics Fundatin Apprach t SROI t Melinda Tuan, 21 May Olsen, Sara. Re: Integrated Cst Appraches t Measuring Scial Impact t Melinda Tuan, 30 April Olsen, Sara and Jeremy Nichlls. a framewrk fr appraches t sri analysis. May Pisell, Tm. What is the Difference Between Internal Rate f Return (IRR) and ROI? [Available Online] 11_gci ,00.html. Prtney, Paul R. Benefit-Cst Analysis. The Cncise Encyclpedia f Ecnmics. [Available Online] Redstne Strategy Grup, LLC. Making Every Dllar Cunt: Hw Expected Return Can Transfrm Philanthrpy. April 10, [Available Online] Dllar+Cunt.htm. Rhdes, H.J., Nnan, K., and Rsqueta, K. Executive Summary. Pathways t Student Success: A Guide t Translating Gd Intentins int Meaningful Impact. Center fr High Impact Philanthrpy, Schl f Scial Plicy & Practice, University f Pennsylvania. Wrking draft. September 2008 (expected publicatin Fall 2008). Rhdes, H.J., Nnan, K., and Rsqueta, K. Hw We Calculated Cst Per Impact in Educatin. Pathways t Student Success: A Guide t Translating Gd Intentins int Meaningful Impact. Center fr High Impact Philanthrpy, Schl f Scial Plicy & Practice, University f Pennsylvania. Wrking draft. September 2008 (expected publicatin Fall 2008). Rsqueta, Kat, Hilary Rhdes, and Kathleen Nnan. Re: Center fr High Impact Philanthrpy Cst Per Impact Measures and ther Integrated Cst Appraches t Measuring Scial Impact t Melinda Tuan, 8 and 24 May Sadwnik, Bryn. Assessing the Perfrmance and Impact f Scial Enterprises: A Survey f Research Prjects. Demnstrating Value Prject. [Available Online] ject_survey_august_2.pdf. August Melinda T. Tuan 41

42 Schlten, Peter. Re: Scial Return n Investment Develpments Since REDF t Melinda Tuan, 16 September Schlten, Peter, Jeremy Nichlls, Sara Olsen and Brett Galimidi. Scial Return n Investment: A Guide t SROI Analysis. Lenthe Publishers, Setel, Philip. Cst-effectiveness; cst-effectiveness; scial return n investment in Glbal Health. Bill & Melinda Gates Fundatin internal mem, April 8, Stut, Susan. Re: Perspectives n Cst-Benefit Analysis and Cst-Effectiveness Analysis in Glbal Health cmmunicatin t Philip Setel, April The New Ecnmics Fundatin. Measuring Real Value: A DIY Guide t Scial Return n Investment. [Available Online] pdf. May, The Rberts Fundatin. SROI Reprts. The Rberts Fundatin [Available Online] Trelstad, Brian. Measuring What Matters. Presentatin t PRI Makers 2008 Cnference. Trelstad, Brian. Re: Acumen Fund Best Available Charitable Optins (BACO) and Prtfli Data Management System (PDMS) t Melinda Tuan, 30 April Trelstad, Brian. Simple Measures fr Scial Enterprise. Innvatins. Vlume 3, Issue 3, Summer 2008, pp Tuan, Melinda and Julia Jnes. SROI Reprts: Overview and Guide. The Rberts Fundatin [Available Online] U.N. Glbal Cmpact. Wh Cares Wins: Cnnecting Financial Markets t a Changing Wrld. December Weinstein, Michael. Measuring Success: Hw Rbin Hd Estimates the Impact f Grants. (wrking draft) January 31, Weinstein, Michael. Re: Rbin Hd Fundatin Benefit-Cst Rati and Integrated Cst Appraches t Measuring Scial Impact t Melinda Tuan, 23 April 2008; cmmunicatins July, August, September Yamada, Tadataka. Analysis f DALYs and Budget $. Bill & Melinda Gates Fundatin cnfidential draft internal dcument Zerbe, Richard O., Jr. and Allen S. Bellas. A Primer fr Benefit-Cst Analysis. Cheltenham: Edward Elgar Publishing Limited Zerbe, Richard O., Jr., Yram Bauman, and Aarn Finkle. An Aggregate Measure fr Benefit-Cst Analysis. Eclgical Ecnmics. 58 (2006) Melinda T. Tuan 42

43 Zerbe, Richard O., Jr. The Legal Fundatin f Cst-Benefit Analysis. Charlestn Law Review. Vlume 2, Fall 2007, Number 1. Melinda T. Tuan 43

44 APPENDIX O: THE FLAT OF THE CURVE One additinal issue that is relevant t the fundatin has nt been addressed by the prfiled appraches: hw t determine where the flat f the curve lies in measuring the csts and impact f a prgram. The term was ppularized by Dr. Alain Enthven, a prfessr emeritus at Stanfrd Graduate Schl f Business t describe the decreasing benefit f increased health care expenditures beynd a certain level f investment/cst. The term was first used in a natinal defense cntext in the 1960s t describe a pint where even small increases in target destructin capability wuld require enrmus increases in cst. 66 The fllwing graph depicts the flat f the curve and shws the relatinship between health and intensity f care at tw different times, t and n years later ( t + n ). In bth perids, pints A and B represent the level f intensity f care that is at the flat f the curve. At this pint, any further health care prvided des nt actually result in an imprvement in health while it des require increased csts. At any given time, plicy usually invlves chsing between mre care r less; gd decisins require cmparing incremental benefit and incremental cst. 67 Anther way t think f the flat f the curve is as the ppsite f a tipping pint it is the pint at which further investment will nt result in greater scial impact. As the scial sectr cntinues t see its grantmaking as having a catalytic rle in addressing majr scial issues, it will be imprtant fr the sectr t cnsider where the flat f the curve lies fr an issue area r a particular prgram (e.g. eradicating malaria thrugh distributin f bed nets). Indeed, there is a benefit t the scial sectr in 66 Enthven, A.C. and K.W. Smith, Hw Much is Enugh: Shaping the Defense Prgram RAND Fuchs, Victr R. Perspective: Mre Variatin In Use Of Care, Mre Flat-Of-The-Curve Medicine: Why des it ccur? What shuld be dne abut it? Health Affairs. 7 Octber Melinda T. Tuan 44