Consolidated Edison s Multifamily Low Income Program: Process Evaluation Summary Evaluation Conducted by: DNV KEMA as subcontractor to Navigant Consulting March 26, 2013 PROGRAM SUMMARY The Multifamily Low Income Program (MFLI) was designed, and subsequently approved, to provide funding to the New York City Housing Authority (NYCHA) and the Westchester County Housing Authorities (WCHA) for prescriptive rebates of up to 100 percent of the incremental cost of qualifying costeffective high efficiency gas heating equipment such as boilers and furnaces. It also provides up to 100 percent of the installed cost for other eligible measures, such as building weatherization measures. Additionally, new technologies or customized applications of other cost-effective energy savings measures may be submitted for program approval. Con Edison administers the MFLI Program and it is implemented through NYCHA and WCHA, with their existing protocols and processes modified to meet the MFLI Program criteria. Con Edison allows custom measures as long as these measures are subject to an independent engineering analysis to estimate energy savings and these measures can pass the Total Resource Cost (TRC) test. Prescriptive program measures include high efficiency water boilers, attic insulation, basement insulation, floor insulation, wall insulation, reduced air infiltration, weather stripping, door sweeps, pipe insulation and HVAC tune-ups and repair. EVALUATION OBJECTIVE AND HIGH LEVEL FINDINGS The final MFLI process evaluation report was submitted on January 10, 2013. The table below summarizes MFLI program savings goals and accomplishments through the 2011 program year. These values have been taken from the monthly scorecard, which is a monthly progress report required by the New York DPS. Energy Savings reported as achieved are ex ante and have not been confirmed by an independent impact evaluation. Program Goal 2009 2011 (Dth) Progress through Year-End 2011 (Dth) 31,350 23,499 75% Percent of Goal Acquired The overall objective of the MFLI process evaluation is to assess the effectiveness and efficiency of program design, delivery and implementation processes. Due to the long time periods required for project implementation in this program, this evaluation has been conducted in two separate phases. The first phase primarily focused on barriers to program participation faced by PHAs, barriers to project implementation faced by participating PHAs, the project identification/approval processes, and interim assessments of program satisfaction and was completed in May of 2011. The second phase focused on project installation, M&V, incentive payment processes and the lessons learned from these processes and was completed in 2012. The MFLI program reached 75 percent of its 2009-2011 energy savings goals by the end of 2011. The reasons for the program not achieving its energy savings goals were mostly due to some NYCHA projects which the program had expected to be implemented in 2011 being delayed until 2012. This program is a traditional resource acquisition program, although it did not have an explicit program theory or logic model. It is a unique program in that it targets PHAs only and works closely with HUD. The PHA s face many barriers to efficiency implementation including multiple approval layers, complex participant requirements and other funding priorities. Program tracking was determined to need improvement; free ridership needs further diligence and program satisfaction varied among PHAs.
A bulleted list of high level findings are provided below. Note that there is also a detailed list of findings later in this document. The involvement of HUD in program implementation complicates customer acquisition. Tracking system was found to need standardization, transparency and documentation Larger PHAs have advantages over smaller ones, making them more likely to participate. The way HUD allocates funds for utility costs is a key barrier to PHA energy efficiency implementation. Program barriers observed in the study include some introduced by HUD and some that are inherent to the multifamily market. Smaller PHAs, in particular, face formidable barriers to energy efficiency implementation, Free ridership is a threat to the MFLI program that will require continued diligence. Con Edison was instrumental in helping move some projects forward. The two participating PHAs were very satisfied with participation while non-participants were not satisfied with program elements encountered (including the project approval process). EVALUATION RECOMMENDATIONS AND PROGRAM ADMINISTRATOR RESPONSE The following recommendations were made by the evaluation team conducting this study. Due to the performance of this evaluation midstream of program operations, Con Edison had already begun to address the issues or had imitated efforts to do so. Recommendation 1: Con Edison or the New York DPS should consider reserving/encumbering a portion of the MFLI incentive dollars for smaller PHAs. The amount of the encumbered dollars could be based on a formula that represents the energy savings potential of the smaller PHAs and to insure that the encumbered funds would not go unused, Con Edison could allow some or all of the encumbered funds to be reallocated for projects submitted by the larger PHAs under certain circumstances. These might include if no small PHAs submitted project proposals before the program deadlines; or if the value of the encumbered funds exceeded the estimated incentives needed for the projects submitted by the smaller PHAs. Response to Recommendation 1: This might be a workable solution if it is integrated into a Commitment Based Approach as a result of HA s submitting viable projects for inclusion in the program. Recommendation 2: Con Edison or the New York DPS should consider working with New York PHAs to introduce an energy-efficiency-based utility allowance program that would help mitigate the reduction in HUD operating cost allocation due to decreased energy consumption over time resulting from the efficiency improvements. Response to Recommendation 2: Under review. Recommendation 3: Make it easier to suggest changes to the multifamily Technical Reference Manual either by adding missing measures (e.g., steam traps) or improving the calculation methods for existing measures. Response to Recommendation 3: Suggesting changes to the Technical Manual is an ongoing issue which is being handled at the Implementation Advisory Group (IAG) level, in concurrence with the PSC s Order on flexibility. This recommendation could be affixed to all EEPS Programs at every Program Administrator, and is not solely attributed to this program alone. Currently, we use the existing IAG process for adding new measures or updating information on existing measures. Recommendation 4: Going forward, require that PHAs and their implementation contractors provide better documentation of saving estimates for the projects they install by requiring that the energy savings estimates be based on the algorithms in the TRM, that end-of-project reports be provided, that all relevant energy savings calculations be available in a single document, that an update to the project application be provided on inspection, and a pre-installation inspection. 2
Response to Recommendation 4: We believe that this recommendation is attainable. Recommendation 5: The program should develop a standard checklist of must have project information to streamline the project approval process. Response to Recommendation 5: This could be problematic due to the customized nature of projects that could be submitted. Due to that a one size fits all approach may not be appropriate, but we will work to develop a list of items that might cover various measure types. Recommendation 6: Make some improvements to the spreadsheets being used to track program information, including the use of a standard format, provision of contact information, use of comment fields, use of ineligible flag measures, simplified file structure and steps to allow for better tracing of savings reported on scorecards. Response to Recommendation 6: Con Edison had issues with support documentation received from the PHA s engineering consultants and so we believe that this recommendation is important for the program moving forward. Recommendation 7: Con Edison should develop more in-house technical resources so it can more effectively mine the energy efficiency opportunities within NYCHA. Response to Recommendation 7: Con Edison has in-house engineers who review project proposals submitted for program inclusion. This, coupled with better support documentation from the PHA s should bring improvements into the program s processes. We are currently meeting with NYCHA on a regular basis, to work on possible projects for the program. Recommendation 8: Con Edison should increase outreach and education efforts to the PHAs about the MFLI program and energy efficiency in general, including the performance of regular listening sessions with the PHAs to find out whether they have any ideas for energy efficiency projects and what these ideas are. Con Edison should also consider developing a Yonkers case study marketing piece, partnering with HUD to educate the PHAs about HUD s energy efficiency opportunities, sponsor lunch and learns with the Westchester PHAs, work with HUD to develop simplified or standardized boilerplate documents that would make it easier for smaller PHAs to participate in the EPC program; and also show smaller PHAs how to do a joint solicitation for an energy performance contractor. Response to Recommendation 8: Con Edison holds annual meetings with representatives from all of the PHA s in NYC and Westchester County. We are currently exploring other options and avenues to enhance the PHA s involvement into the program, however the size of our effort needs to be commensurate with the program s scope and budget. Recommendation 9: Con Edison should adopt some practices to try to reduce program free ridership by getting the program more involved in projects at a very early stage. These practices might include educating smaller PHAs about energy efficiency opportunities, Connecting PHAs to energy audit and other technical resources, helping PHAs sell energy efficiency projects to their boards of directors, facilitating their participation in the HUD s EPC program; and helping them attract energy performance contractors and other funding sources for capital improvements. Response to Recommendation 9: The second EEPS cycle (2012-2015) has allowed us to work with PHA s as they develop programs that can be designed specifically for program inclusion. DETAILED PROCESS EVALUATION FINDINGS Detailed key findings include: Con Edison did not develop an explicit program theory or logic model for the MFLI program, however, our assessment is that it is a traditional resource acquisition program. The involvement of HUD complicates customer acquisition. One unique feature of the MFLI program is the involvement of HUD. For MFLI projects to be funded they must receive HUD approval, in addition to Con Edison approval. The HUD approval process introduces an additional complication in implementing projects that other Con Edison energy efficiency programs do not have, and there are other programs available to the PHA s.
Tracking system was found to need standardization, transparency and documentation Larger PHAs are noted to have advantages over smaller ones that make them more likely to participate. The way HUD allocates funds for utility costs is a key barrier to energy efficiency implementation for PHAs. HUD bases the PHA s budget for operating costs on a three-year historical average. Since utility costs are basically a pass through cost to HUD, there is limited incentive for PHAs to reduce them. Program barriers introduced by HUD include the manner it allocates energy efficiency program funding or compensates PHAs for utility costs. Other barriers include structural barriers, such as multiple layers of decisions makers and the fact that tenants do not pay their own energy bills. Smaller PHAs, in particular, face formidable barriers to energy efficiency implementation. These include having no economies of scale for energy savings, having insufficient upfront capital, lacking necessary technical knowledge, having insufficient time or sophistication to consider larger energy efficiency projects, and having difficulty attracting performance contractors. Free ridership is a threat to the MFLI program that will require continued diligence. The first come, first served MFLI approach presents an advantage to energy efficiency projects that are already far along in their design and development cycle. Such projects are at greater risk of low program attribution (high free ridership) because program involvement is at a later stage, when funding and approvals are already secured. Con Edison was instrumental in helping move some projects forward. The two participating PHAs were very satisfied with participation while non participants were not satisfied with program elements encountered (including the project approval process). Reasons for dissatisfaction included uncertainty around incentives, irregular pace of the project work and communication, and limited PHA participation. EVALUATION METHODS AND SAMPLING The research and the findings expressed in the report and this summary are based upon the following evaluation activities: Review of program planning and marketing materials, Review of program tracking system, data, and other program delivery documents, In-depth interviews with program implementers including: Con Edison MFLI program managers (three full interviews completed, July 2010 (first program manager), March 2011 (first program manager), and May 2012 (second program manager); and ICF International (two interviews completed, February 2011 (one ICF representative) and February 2012 (two ICF representatives). In-depth interviews with participating PHAs including: o Three interviews with a representative of the Yonkers PHA (July 2010, March 2011, and May 2012); o Four interviews with three representatives of NYCHA (July 2010, February 2011, March 2011); and o One interview with a representative of the Town of Mamaroneck (April 2012). o The Yonkers PHA implementation contractor Honeywell (2 interviews March 2011, April 2012); In-depth interviews with nonparticipating PHAs (5 interviews January-March 2011); and In-depth interview with Housing and Urban Development (1 interview in February 2011). 4
The information from the various data collection activities were synthesized to inform the primary objectives of the study with a particular interest in identifying where changes in various elements of program delivery might make a big difference in program efficacy and participation rates. The data gathered were qualitative in nature and the purpose of the analysis was to relate what we have learned and what it means to Con Edison and the DPS. Of particular importance to this exercise was bringing together of the knowledge of program operations from our data review and program implementer interviews, PHA program knowledge and experiences with the program, and the reported needs and concerns of non-participating PHAs.