Cnslidated Edisn f New Yrk: Residential Direct Install Prgram: Prcess Evaluatin Summary Evaluatin Cnducted by: DNV KEMA as subcntractr t Navigant Cnsulting PROGRAM SUMMARY March 13, 2013 Cn Edisn designed the Residential Direct Installatin (RDI) prgram as an entry pint fr residential custmers t evaluate their hme s energy perfrmance and identify energy savings pprtunities. The target market is residential custmers defined as single family hmes and multi-family hmes with 2 t 4 units. The prgram ffers a hme energy survey fr $50. The purpse f the survey is t evaluate the custmer s hme fr energy perfrmance, dcument existing equipment and then identify and recmmend energy efficiency upgrades. Fllwing the energy survey, the prgram installs up t nine lw-cst, energy efficient measures. The energy efficiency measures include: six cmpact flurescent lamps (CFLs), ne smart strip, ht water pipe insulatin, lw-flw shwerheads, weather stripping, dr sweeps, windw air cnditiner timers, faucet aeratrs and setting back the water heater temperature. The prgram allts up t 10 CFLs per participant. EVALUATION OBJECTIVE AND HIGH LEVEL FINDINGS Table ES1 summarizes the RDI prgram savings gals and reprted achievements fr Cn Edisn frm prgram initiatin in 2010 thrugh July 2011 (when the evaluatin team fielded its surveys f participants and nn-participants) 1. The table als cntains cumulative prgram achievement infrmatin as f 2011 year-end. The prgram achieved rughly ne-third f its survey gal (34%); at 20%, it fell shrt f its energy savings gals. Parameter Table ES1. Cn Edisn RDI Gals and Reprted Achievements Prgram Gal 2010 2011 Prgress thrugh July July Percent f 2011 Gal Achieved Prgress thrugh Year-End 2011 Year-End Percent f Gal Achieved Number f Surveys 8,252 1,848 22% 2,832 34% Savings (MWh) 6,425 868 13% 1,295 20% Cincident Peak Savings (MW) 1.56 0.05 3%.13 9% The verall bjective f the RDI prcess evaluatin is t assess the effectiveness and efficiency f prgram design, delivery and implementatin prcesses. Specifically, the evaluatin addressed six key prgram prcesses: prgram planning, infrastructure develpment, marketing and custmer acquisitin, prgram delivery, satisfactin with the prgram and interactins with ther prgrams. The RDI prgram is intended t serve as an entry pint fr residential custmers wishing t identify pprtunities fr imprved efficiency in their hme. Cn Edisn cntracts with Hneywell t deliver the RDI prgram n their behalf. The RDI prgram did nt achieve its 2010-2011 prgram gals. It achieved 20 percent f its MWh savings gal and 9 percent f MW savings gal by the end f the year. The RDI Prgram generated its highest ttals f savings by mnth in its first few mnths f peratin (Octber 2010 thrugh December 2010). The prgram psted lwer mnthly savings in 2011. 1 It is imprtant t pint ut that achieved energy savings reprted in Table ES1 are ex ante savings estimates savings estimates which have nt been cnfirmed by an independent impact evaluatin.
Althugh the prgram perated reasnably well, the achievement f prgram gals was greatly hindered by its inability t generate new custmer interest as well as never achieving the average savings per participating hme necessary. Prgram satisfactin levels were generally high amng participants and prgram tracking was deemed sufficient. There are several areas fr imprvements nted belw, all intended t imprve the chance f prgram success. Hwever, the cre issues hlding back prgram success are related t marketing and the lack f sufficient measure installatin quantities at participating hmes. EVALUATION RECOMMENDATIONS AND PROGRAM ADMINISTRATOR RESPONSE The fllwing recmmendatins were made by the evaluatrs cnducting this study. Cnslidated Edisn s initial respnse t these recmmendatins is als summarized belw and will be tracked ver time. Recmmendatin 1: Prgram Planning and Design a) Cmmunicate and fcus n increasing the number f installed measures per visit, r review prgram fferings and cnsider measure mix ptins that might have better installatin rates than thse currently prvided. b) Market aggressively t prgram-eligible custmer grups which currently have lw prgram participatin cmpared t their ppulatin. c) Aim t and make any needed staffing changes t cmplete mre hme energy surveys thrugh increased marketing and subsequent custmer prgram demand. Respnse t Recmmendatin 1: Since this research was cnducted, Cn Edisn s Marketing effrts have been centralized, greatly enhanced and better aligned, as a result f ur full time Marketing Sectin Manager cming n bard. Staffing changes are directly related t vlume, and ur Implementatin Cntractr has said that when demand increases, they wuld have n issue with increasing staff. Recmmendatin 2: Infrastructure and Develpment a) Cnslidated Edisn shuld revisit quality cntrl prcesses regarding savings calculatins and the manner in which values are transferred frm the database t the screcard reprts. Cnslidated Edisn shuld als cnsider adding a measure-level identificatin (ID) field t bth the custmer installs table in the database and the screcard reprts and cnsider tracking multiple phne numbers in separate fields, either by type f phne (hme, wrk, mbile) r rder f preference (phne number 1, phne number 2, etc). Respnse t Recmmendatin 2: While this prgram cntinues t perate, the prgram s future remains in dubt, and will be assessed in the future. Recmmendatin 3: Marketing and Custmer Acquisitin a) Develp an agreement between Cn Edisn and Hneywell abut the rles that each entity will be respnsible fr in marketing the RDI Prgram. b) Target future marketing in a manner that will allw technicians t wrk mre efficiently in a smaller gegraphic area. c) Prvide clear and direct cmmunicatin abut specific prgram fferings thrughut the prgram s website and direct mail materials. d) Streamline website custmer call-t-actin effrts t reduce custmer cnfusin, and increase the likelihd f cntracting the prgram thrugh the website. 2
e) Ensure Cn Edisn has fully capitalized n the pprtunity t market the RDI prgram at naturally ccurring tuch pints such as custmer call centers. f) Create a baseline n number f page views, time n page, and click-thrugh rates as a basis fr regular review f web activity befre undertaking significant website revisins. g) Cntinue t emphasize saving energy and mney in RDI marketing messages. h) As prgram activity increases, cnsider autmating custmer infrmatin / survey requests between the Cn Edisn website and the Hneywell prgram schedulers. i) Energy survey technicians shuld cntinue t request participants tell friends/family members if they are satisfied with the prgram. Respnse t Recmmendatin 3: Any website revisins will be cntingent upn the cntinuatin f the prgram. Individual prgram marketing plans are currently being develped in 2013. These plans shuld be ready smetime in the latter prtin f Q1 f 2013. Hwever, a specific marketing plan and funds fr this prgram will be limited and embedded in crss marketing effrts, as benefits frm the ther residential prgrams culd enhance this prgram as well. Recmmendatin 4: Prgram Delivery a) Cn Edisn and Hneywell shuld review the newly implemented HmeSTAR survey reprt tracking lg and custmer service curtesy call prcedures in the first half f 2012 t assure that it is functining prperly. b) Regularly review the capacity f prgram staffing levels t meet prgram gals and cnsider hiring additinal staff as apprpriate. c) Clearly message the need fr Hneywell t increase the number f installed measures per visit, whenever pssible, and mnitr whether the prgram change t install mre smart strips r CFLs translates int additinal prgram savings. d) Cntinue t emphasize t energy survey technicians that prgram measures must be installed at participant hmes. e) Implement prgram changes and marketing material updates simultaneusly. f) Cnsider emailing Hme Energy Survey participants their results. g) Cn Edisn branding in the field shuld be strengthened thrugh imprved use f Cn Edisn prgram identificatin in the field. h) Cnsider streamlining prgram payment prcesses by ffering custmers mre payment ptins (e.g., PayPal, Square, etc.). Respnse t Recmmendatin 4: While the prgram remains peratinal, any revisins made will be cntingent upn the cntinuatin f the prgram, and the status f the petitin filed by the Cmpany. Recmmendatin 5: Prgram Satisfactin a) We recmmend that Cn Edisn verify custmer receipt f the HmeSTAR reprt as part f the nging Hneywell nline survey effrt amng participants. Respnse t Recmmendatin 5: While the prgram remains peratinal, any revisins made will be cntingent upn the cntinuatin f the prgram, and the status f the petitin filed by the Cmpany. Recmmendatin 6: Interactins with Other Prgrams
a) Cn Edisn shuld ensure that all pprtunities t brker ther prgrams as an integral part f the hme energy survey are fully leveraged. Energy survey technicians shuld be prmting additinal Cn Edisn energy efficiency prgrams during every Hme Energy Survey. Imprving RDI prgram satisfactin may be the prgram s strngest marketing and prmtinal asset. Respnse t Recmmendatin 6: While the prgram remains peratinal, any revisins made will be cntingent upn the cntinuatin f the prgram, and the status f the petitin filed by the Cmpany. DETAILED PROCESS EVALUATION FINDINGS Cnslidated Edisn was nt successful in meeting its savings targets. The average achieved savings per visit is cnsiderably less than what was anticipated in planning. The study als nted pprtunities t bst prgram awareness and participatin within the brughs (such as Queens and Brklyn) with the largest prgram-eligible custmer ppulatin (e.g., Brklyn) and amng residents f multi-family (2-4) buildings. Prgram peratins were nted t be generally successful at addressing energy efficiency barriers as utlined in the prgram s lgic mdel. Thrugh increased marketing and subsequent custmer prgram demand, and ultimately, by executing mre hme energy surveys, the prgram will increase its energy savings achievements. The BBCS database appears t be sufficient fr implementatin and evaluatin needs, and cmprehensively tracks cmplete infrmatin fr each site, including measures installed during the survey, recmmended measures, and tips fr behavir changes t save energy. Hwever, savings ttals frm the database d nt match thse in the screcard reprts, savings in the database d nt use the same per unit savings values fr measures f the same type and several recrds have greater than expected measures savings and/r quantities. The Pwer f Green (Green Team) marketing campaign utilized many marketing vehicles t cmmunicate Cn Edisn s suite f available energy efficiency prgrams and emphasize its green brand. The vast majrity f RDI prgram participants indicated they initiated prgram cntact by phne and reprted that saving mney and energy was a majr reasn fr prgram participatin, which is cnsistent with the appeal in current prgram marketing. The lack f leads has resulted in less than ptimal survey schedules. Green Team materials prmting the RDI prgram (including the website) are generally adequate, but d nt prvide infrmatin that implementers believe are mre apprpriate fr presentatin in-persn and n-site, such as prgram participatin fees, and ften d nt list measures ther than CFLs and smart strips. Only abut ne-quarter f RDI participants and even fewer nn-participants (10%) reprt hearing abut the prgram thrugh the Cn Edisn website r using it in the past year. Bth RDI prgram participants and nn-participants mst frequently mentin hearing abut the RDI prgram thrugh a Cn Edisn bill insert. Wrd-f-muth was nt frequently named as a surce f prgram awareness, althugh, field technicians were bserved asking participants t talk t their family and/r friends abut the prgram if they agreed it was a psitive experience. Hneywell is generally servicing custmers wh request a Hme Energy Survey prmptly. Hneywell has internal meetings that are effective and prvide pprtunity fr best practices t be shared and fr QA/QC issues t be reslved and emphasized as necessary. Technicians were bserved t cnsistently perfrm the survey elements fr each participant with the exceptin f sme leave behind materials, which differed slightly. The technicians were able t answer mst custmer questins directly and adequately, r prperly referred the questin t the quality cntrl crdinatr. Nearly three-furths f participants recalled the energy survey technician discussing the energy survey results with them after cmpleting the survey and nearly tw thirds were able t recall receiving specific recmmendatins. A majrity f participants indicated they received a written reprt f their results in three weeks r less, althugh fewer than 54% f them reprted that the recmmendatins in the reprt were helpful. Installed measure satisfactin rates were reasnably 4
high, with the tp three mst frequently installed rated a 7 r higher. Hwever, twenty-nine (13%) participants wh received a smart strip indicated the technician did nt plug equipment int the smart strip. The average amunt f measure activity installed was significantly shrt f the average necessary t meet verall prgram savings gals. Participant satisfactin with the prgram is high, with fifty-five percent ranking their satisfactin at 8 r higher n a 10-pint satisfactin scale, althugh twenty percent f participants rated their prgram satisfactin lwer than 5. This latter grup mst cmmnly indicated the prgram was nt helpful t them, lacked infrmatin r were frustrated by nt recalling having received a written reprt frm the prgram. Energy survey technician ratings were very high and mst participants strngly agreed that energy survey technicians explained the prgram and answered their questins. Nearly three ut f fur (73%) participants wuld recmmend the prgram t a friend r family member. Rughly ne ut f every five prgram participants recalled receiving Cn Edisn prgram recmmendatins frm their energy survey technician and sixty-five percent f participants indicated their survey technician did nt refer them t ther Cn Edisn energy efficiency prgrams. Nearly tw-thirds f prgram participants wh cnfirmed receipt f prgram recmmendatins reprted a referral t the Appliance Bunty prgram. RDI participants were mre likely t participate in ther Cn Edisn energy efficiency prgrams (Appliance Bunty, Res HVAC, and/r Rm AC) cmpared t nn-participants althugh rughly tw-thirds f bth RDI prgram participants and nn-participants indicated n awareness f ther energy efficiency prgrams utside f Cn Edisn fferings. EVALUATION METHODS AND SAMPLING The research and the findings expressed in this reprt are based upn the fllwing evaluatin activities: Review f prgram planning and marketing materials, Review f prgram tracking system, data, and ther prgram delivery dcuments, In-depth interviews with: Cn Edisn staff (tw cmpleted, June 2010 and February 2011, respectively) Hneywell staff (six cmpleted, July 2010 t August, 2011) Hneywell subcntracted energy survey technicians (fur cmpleted, August, 2011 t Octber, 2011) Prgram participants (five cmpleted, May and June 2011) Custmer telephne surveys with: Prgram participants (293 cmpleted, Octber and Nvember 2011) Prgram nn-participants (297 cmpleted, Nvember and December 2011), and Field and ffice bservatins f implementatin staff and energy survey technicians (August 18 th and 19 th, 2011).