Final Report Rocky Mountain Power ENERGY STAR New Homes Program Evaluation Prepared for: Rocky Mountain Power March 28, 2011
Prepared by: Brian Hedman Sara Wist Cadmus Group K:\2009 Projects\2009-338 (PC) PC Multi Pgm Evaluation {6368.0001}\Residential\Energy Star New Homes\Reporting\ES New Homes Evaluation 032510.docx
Table of Contents 1. Executive Summary...1 Program Summary...1 Eligibility Requirements and Incentives Program Assumptions and Participation Evaluation Approach Impact Evaluation Results...2 Freeridership Results Spillover Results Process Evaluation Results...4 Best Practices Review Interview and Survey Results Cost-Effectiveness Results...5 Conclusion and Recommendations...6 Correct Sizing of Air Conditioning Units Program Data Collection 2. Program Description...9 Eligibility Requirements and Incentives...9 Measures and Incentives Cooperative Marketing Program Assumptions and Participation...10 Program Logic and Progress Indicators...12 3. Impact Evaluation Methodology... 15 Billing Analysis...15 Home Sample Selection and Data Collection Billing Data Weather Matching and Data Screening Sample Matching Energy Analysis PRISM-like Modeling Usage Per Square Foot Distribution Usage Per Square Foot Savings Estimation Model Freeridership and Net-to-Gross...20 1 2 2 3 3 4 4 7 7 9 10 15 15 15 16 17 19 The Cadmus Group, Inc. / Energy Services i
Spillover...20 Air Conditioner Site Visits...20 4. Process Evaluation Methodology... 21 Materials and Best Practices Review...21 Interview and Survey Data Collection...21 Utility and Staff Interviews...22 Builder and Homeowner Surveys...22 Participant and Nonparticipant Builders Participant and Nonparticipant Homeowners 5. Impact Evaluation Results... 25 Gross Energy Savings Estimates...25 Freeridership Results...26 Spillover Results...26 Air Conditioner Site Visit Results Key Findings...27 6. Process Evaluation Materials and Best Practices Review Results... 29 Materials Review...29 Rocky Mountain Power s ENERGY STAR New Homes Program Web Site Application and Inspection Materials Program Reports and Data Best Practices Review Results Key Findings...34 7. Process Evaluation Interview Results... 35 Utility Staff and Implementer Interviews...35 ES New Homes Implementation and Field Work Key Findings...41 8. Builder Survey Results... 43 Participant Builder Survey Results...43 Program Participation and Awareness Coordination with Questar Application Process Building Practices and Freeridership 22 23 27 29 29 30 30 36 43 45 46 49 The Cadmus Group, Inc. / Energy Services ii
Building Codes Marketing Program Satisfaction Nonparticipant Builder Survey Results...64 Program Awareness and Participation Decisions Energy-Efficient Building Practices Current Building Practices Key Findings...75 9. Process Evaluation Homeowner Survey Results... 77 Participant Homeowner Survey Results...77 Program Awareness Influence of Industry Workers Decision to Participate Participation in Other Utility Programs Energy Expectations Nonparticipant Homeowner Survey Results...90 Awareness and Purchase Information Participation Decision Key Findings...99 10. Cost-Effectiveness Analysis... 101 11. Conclusion and Recommendations... 105 Correct Sizing of Air Conditioning Units...105 Program Data Collection...105 Appendix A: Data Request Form... 106 Appendix B: Utility Staff Discussion Guide... 107 Appendix C: Implementer Staff Discussion Guide... 108 Appendix D: Participant Builder Discussion Guide... 109 Appendix E: Nonparticipant Builder Discussion Guide... 110 Appendix F: Participant Homeowner Guide... 111 54 55 62 65 66 70 77 78 80 87 87 90 95 The Cadmus Group, Inc. / Energy Services iii
Appendix G: Nonparticipant Homeowner Guide... 112 Appendix H: On-Site Collection Form... 113 Appendix I: Manual J Assumptions... 114 The Cadmus Group, Inc. / Energy Services iv
1. Executive Summary Program Summary Rocky Mountain Power implements the ENERGY STAR New Homes program (the program) in Utah. The program promotes construction of energy-efficient homes that save money and energy, and conserve natural resources. ENERGY STAR-qualified homes are independently verified to be at least 15% more efficient than the state energy code. These savings are based on heating, cooling, and hot water energy use, and are typically achieved through a combination of measures such as building envelope upgrades, high-efficiency windows, upgraded HVAC, and weatherization. The program provides incentives to builders constructing ENERGY STAR homes, with the incentive amount varying, depending on specific efficiency tiers and the installation of additional efficiency measures. The program has separate requirements for single-family and multifamily homes. This evaluation assessed the program for each year, 2006 through 2008, relying primarily in billing data analysis to measure energy savings. Eligibility Requirements and Incentives The program is open to all residential new construction of five stories or less within the utility s service territory, billed on Schedules 1, 2, and 3. Program measures and incentives, which have evolved to reflect building code and market changes, are designed with a whole house approach to energy savings. Each year of the program, homes are required to meet that current year s ENERGY STAR specifications, which include combinations of the following: Performance-based duct sealing Air conditioner equipment minimum standards A/C performance testing Correct sizing Best practice installations Thermal bypass checklist Installation of CFLs In addition to ground source heat pump options, plus measures are available to builders, and include incentives for: Duct placement inside of conditioned space Single vent evaporative cooling systems Ducted high-efficiency evaporative cooling systems Whole house fan systems ENERGY STAR dishwashers, ceiling fans, and lighting upgrades The Cadmus Group, Inc. / Energy Services 1
Incentives, ENERGY STAR specifications, and plus measures have been adjusted annually as necessary to reflect changes in standards and the market. Program Assumptions and Participation The method by which energy savings has been attributed to the program has also evolved within the evaluation period. Prior to mid-2008, deemed savings estimates for single-family and multifamily construction were calculated by averaging savings values attributed to building simulations modeled to represent homes at each of the available tier levels. The deemed savings value for each construction type was attributed to all homes of that construction type, regardless of tier level or addition of plus measures. An improvement to the estimates was implemented beginning in June 2008, when savings estimates for the program were expanded to include annual savings estimates attributable to each of the tier levels as well as for the individual Plus measures. Table 1 presents total homes and estimated program savings for the program during the evaluation period. Table 1. Program Assumptions and Participation 2006 2008 Program Year Single-Family Homes Multifamily Homes Total Homes 1 Total Savings (kwh/year) 2006 1,758 0 1,758 3,394,698 2007 2,184 88 2,272 3,604,560 2008 1,612 232 1,844 2,422,497 Total 5,554 320 5,874 9,421,755 Evaluation Approach This evaluation s primary objectives are to assess energy savings, estimate freeridership, assess perception and awareness of the program, and collect data necessary to perform costeffectiveness analysis. The evaluation consisted of: Analysis of billing data Interviews with utility and implementation staff Surveys with 21 participating and 20 nonparticipating builders Surveys with 101 participant and 102 nonparticipant homeowners Assessment of freeridership Verification of air conditioner sizing Calculation of cost-effectiveness Impact Evaluation Results The program is generally well received by builders as well as homeowners living in ENERGY STAR-certified homes. Table 2 illustrates participation and estimated savings resulting from the billing data and freeridership analyses compared to that reported in the DSM goals and annual report. 1 Total homes reported in DSM Goals and Annual Report. These totals vary from evaluated totals. The Cadmus Group, Inc. / Energy Services 2
Table 2. Summary of Participation and Savings for ES New Homes Program 2006 2007 2008 Total Reported Participation (Homes) 1,758 2,272 1,844 5,874 Verified Participation (Homes) 1,800 2,290 1,844 5,934 Estimated Energy Savings (kwh) 3,394,698 3,604,560 2,422,497 9,421,755 Evaluated Savings (kwh) 3,573,078 3,354,584 2,054,639 8,982,301 Net Evaluated Savings (kwh) 2,658,370 2,495,811 1,513,897 6,668,077 Table 3 provides summaries of program costs. Table 3. Summary of Program Costs Single-Family Costs 2006 2007 2008 Administration $107,126 $52,198 $28,946 Incentives $707,900 $810,664 $688,801 Implementation $1,033,130 $1,041,341 $867,296 Multifamily Costs 2006 2007 2008 Administration 0 $882 $1,435 Incentives 0 $13,706 $34,159 Implementation 0 $17,606 $43,011 Total $1,848,156 $1,936,397 $1,663,648 Freeridership Results The percent of savings that would have occurred in the program s absence freeridership was calculated through surveys with program participants. In calculating freeridership, Cadmus conducted telephone surveys with 20 builders representing 19 companies responsible for 1,185 homes through the ESNH program in years 2006 through 2008. Given the relatively small number of builders surveyed, a single freeridership value of 26% was calculated for the threeyear evaluation period, resulting in a net-to-gross (NTG) ratio of 74%. Spillover Results As a qualitative assessment of the spillover potential among participant builders, we asked if additional energy-efficiency technologies had been installed in any of their other new homes since they began participating in the Rocky Mountain Power program, but for which they received no incentives. Approximately half of the builders surveyed indicated additional energyefficiency measures or technologies had been installed without direct benefit from the program. Most of these respondents indicated that the program had been influential in this decision. Additional measures and technologies identified by builders providing examples included: Zero Net Energy construction Insulation Furnaces The Cadmus Group, Inc. / Energy Services 3
Process Evaluation Results Best Practices Review A review of program materials as well as benchmarking guidelines was conducted. It was found that program materials were generally clear and easy to navigate. Potential improvements identified include: Develop specific language regarding the definition of correct sizing for AC units. Include space on the builder application for a narrative or documentation of a correct sizing method. Develop verification requirements for correct sizing, with consideration given to a pass/fail screen on the inspection form. Revisit the method of tracking the program year in program data. Audit participation data to ensure tracking is accurate, and consider revisiting the design of the database to ensure accurate, replicable data tracking. The best practices benchmarking table indicates the program is meeting many of the benchmarks. Potential opportunities for further improvements may be worth exploring in the area of program theory and design, management, implementation, and evaluation. Interview and Survey Results Utility and Implementation Staff Interviews were conducted with utility and implementation staff, all of whom consider the program a success. While it was reported that the program is dynamic in its response to changing market conditions, these adjustments sometimes can be seen as barriers by builders navigating the program changes. Potential improvements that could be considered include: Availability of electronic inspection form submission Coordination with Building Code Officials Builders The surveys revealed a high level of program satisfaction among builders participating in the program. These builders had numerous reasons for participating, with most being interested in building better homes. While most builders felt the application process was very easy, some reported experiencing challenges from working with two different utilities, gas and electric. Confusion over incentives and measure descriptions, as well as long incentive processing times was noted, though by a minority of respondents. While ENERGY STAR brand recognition was thought to be high, and there was an expectation these homes save homeowners money, nonparticipating builders most frequently cited costs as the reason for not participating. They also reported a perception that first costs prevented homeowners from pursuing ENERGY STAR The Cadmus Group, Inc. / Energy Services 4
homes. However, many builders reported ENERGY STAR homes sold faster and at or above the price of conventional homes. Homeowners Most homeowners purchasing homes incented through the program were aware that their homes were ENERGY STAR certified, having learned this from their realtor or agent. ENERGY STAR certification was reported as being an important factor in the homebuyers decisions, even among those who unaware of their own homes certified status. Conversely, less than half of homeowners living in conventional homes were aware of the option to buy ENERGY STAR certified homes. Cost-Effectiveness Results Program benefits are comprised of energy savings and their associated avoided costs. The energy savings used in the cost-effectiveness analysis are the evaluated kwh from this study. Table 5, Table 6, and Table 7 present the results of the cost-effectiveness analysis for the Program in 2006, 2007, and 2008 respectively. Table 8 shows the cost-effectiveness analysis for all program years combined (2006-2008). All analyses are based on the Rocky Mountain Power 2008 IRP 46% load factor (LF) eastside residential whole house decrement. The program was found to be cost-effective under both the Utility Cost Test and the Participant Cost Test in 2006 and 2007. The program was not cost-effective under the Utility Cost Test in the year 2008 due to lower than expected savings for that year, which are discussed in the Impact Evaluation Results. Table 4. Cost-Effectiveness Summary for 2006 IRP 46% LF Eastside Residential Whole House Decrement Cost Effectiveness Test Levelized $ / kwh Costs Benefits Net Benefits Benefit / Cost Ratio Total Resource + Conservation Adder (PTRC) $0.103 $4,048,012 $3,674,517 -$373,495 0.91 Total Resource No Adder (TRC) $0.103 $4,048,012 $3,340,470 -$707,542 0.83 Utility (UCT) $0.047 $1,848,156 $3,340,470 $1,492,314 1.81 Ratepayer Impact (RIM) $0.143 $5,630,431 $3,340,470 -$2,289,961 0.59 Participant (PCT) $0.074 $2,907,756 $4,490,175 $1,582,419 1.54 Lifecycle Revenue Impact ($/kwh) $0.00000539 Discounted Participant Payback (Years) 13.99 The Cadmus Group, Inc. / Energy Services 5
Table 5. Cost-Effectiveness Summary for 2007 IRP 46% LF Eastside Residential Whole House Decrement Cost Effectiveness Test Levelized $ / kwh Costs Benefits Net Benefits Benefit / Cost Ratio Total Resource + Conservation Adder (PTRC) $0.113 $4,153,114 $3,510,821 -$642,293 0.85 Total Resource No Adder (TRC) $0.113 $4,153,114 $3,191,655 -$961,458 0.77 Utility (UCT) $0.053 $1,936,397 $3,191,655 $1,255,259 1.65 Ratepayer Impact (RIM) $0.146 $5,384,531 $3,191,655 -$2,192,875 0.59 Participant (PCT) $0.083 $3,041,088 $4,272,505 $1,231,417 1.40 Lifecycle Revenue Impact ($/kwh) $0.00000507 Discounted Participant Payback (Years) 14.72 Table 6. Cost-Effectiveness Summary for 2008 IRP 46% LF Eastside Residential Whole House Decrement Cost Effectiveness Test Levelized $ / kwh Costs Benefits Net Benefits Benefit / Cost Ratio Total Resource + Conservation Adder (PTRC) $0.173 $3,280,136 $1,658,379 -$1,621,757 0.51 Total Resource No Adder (TRC) $0.173 $3,280,136 $1,507,618 -$1,772,518 0.46 Utility (UCT) $0.088 $1,663,648 $1,507,618 -$156,031 0.91 Ratepayer Impact (RIM) $0.180 $3,425,634 $1,507,618 -$1,918,017 0.44 Participant (PCT) $0.123 $2,339,448 $2,484,946 $145,499 1.06 Lifecycle Revenue Impact ($/kwh) $0.00000516 Discounted Participant Payback (Years) 18.44 Table 7. Cost-Effectiveness Summary for 2006-2008 IRP 46% LF Eastside Residential Whole House Decrement Cost Effectiveness Test Levelized $ / kwh Costs Benefits Net Benefits Benefit / Cost Ratio Total Resource + Conservation Adder (PTRC) $0.119 $10,758,667 $8,381,161 -$2,377,506 0.78 Total Resource No Adder (TRC) $0.119 $10,758,667 $7,619,237 -$3,139,430 0.71 Utility (UCT) $0.057 $5,093,424 $7,619,237 $2,525,814 1.50 Ratepayer Impact (RIM) $0.151 $13,613,797 $7,619,237 -$5,994,559 0.56 Participant (PCT) $0.086 $7,767,481 $10,622,610 $2,855,129 1.37 Lifecycle Revenue Impact ($/kwh) $0.00001484 Conclusion and Recommendations Rocky Mountain Power s ENERGY STAR New Homes program has demonstrated energy savings and high levels of staff and participant satisfaction. Overall, it has succeeded at reaching builders and furthering the ENERGY STAR brand, and has been well received by builders as well as homeowners. However, freeridership does appear to be impacting program attributable savings, and the correct sizing of air conditioning (AC) units installed through the program was The Cadmus Group, Inc. / Energy Services 6
identified as a potential issue 2. While findings are presented in each results chapter of this report, a summary of key recommendations follow below. Correct Sizing of Air Conditioning Units Consideration of the following is recommended: Program materials should offer an explanation of the term correct sizing, including the sizing method required by the program. The builder application form should be modified to include a description of the correct sizing method, as well as a signature box for the builder to indicate agreement with the method, and a commitment to provide compliance documentation. The HERS rater form should be modified to include fields for verification of methods used to size the AC equipment, along with a field to indicate whether the home passes or fails based their on this criteria. 3 Survey responses indicate implementation of these changes may result in a reduction of builders participating in the program and homes incented through the program. Program Data Collection While the overall data collection process appears to be smooth, consideration of returning to an electronic method of submitting HERS inspection forms would reduce processing times and streamline the process. Additionally, refinements of the current system may be useful in ensuring consistent data tracking. 2 This issue was explored in depth through a study commissioned by Rocky Mountain Power and conducted by Ecos Consulting, in partnership with Ecotope, Inc and Delta T. ENERGY STAR for New Homes Final Report Air Conditioning Sizing Audit, January 17, 2011 (Ecos Study). 3 The Ecos Study further reviewed the implementation/verification process utilized by the program and developed recommendations to improve the verification process. These recommendations, and others, are found on Page 21 of the report. The Cadmus Group, Inc. / Energy Services 7
2. Program Description Rocky Mountain Power contracts the implementation of the ENERGY STAR New Homes program (ES New Homes) in Utah. This program promotes construction of energy-efficient homes that save money and energy, and conserve natural resources. ENERGY STAR-qualified homes are independently verified to be at least 15% more efficient than the state energy code. These savings are based on heating, cooling, lighting, and hot water energy use, and are typically achieved through a combination of measures such as building envelope upgrades, high-efficiency windows, upgraded HVAC, and weatherization. The program provides incentives to builders constructing ENERGY STAR homes, with the incentive amount varying, depending on specific efficiency tiers and the installation of additional efficiency measures. The program has separate requirements for single-family and multifamily homes. Eligibility Requirements and Incentives The program is open to all residential new construction of five stories or less within the utility s service territory, billed on Schedules 1, 2, and 3. Measures and Incentives Program measures and incentives, which have evolved to reflect building code and market changes, are designed with a whole-house approach to energy savings. In each year of the program, homes are required to meet the current year ENERGY STAR specifications, which include combinations of the following: Performance-based duct sealing Air conditioner equipment minimum standards A/C performance testing Correct sizing Best practice installations Thermal bypass checklist Installation of CFLs In addition to ground source heat pump options, plus measures are also available to the builders, and include incentives for measures such as: Duct placement inside of conditioned space Single vent evaporative cooling systems Ducted high-efficiency evaporative cooling systems Whole house fan systems ENERGY STAR dishwashers, ceiling fans, and lighting upgrades Incentives, ENERGY STAR specifications, and plus measures have been adjusted annually as necessary to reflect changes in standards and the market. In addition to these annual revisions, an adjustment to program measures and incentives occurred in mid-year 2008. Incentives paid each The Cadmus Group, Inc. / Energy Services 9
year to participating homes during the evaluation period are presented in Table 9, with both 2008 incentive amounts listed when applicable. Table 8. Program Incentives Paid to Incented Homes, 2006-2008 Single-Family 2006 2007 2008 Tier 1 $350 $250 $250 $200 Tier 2 $550 $350 $250 $300 Tier 3 NA NA $500 Tier 4 NA NA $700 GSHP Option NA NA $2,000 14 SEER HVAC NA NA $100 ENERGY STAR Dishwasher $10 $10 $10 Duct within Conditioned Space $200 $200 $200 $100 Ducted Evaporative Cooling NA NA $750 ENERGY STAR ceiling fan NA NA $10 Lighting $50 $50 $50 Whole House Fan System NA NA $100 Multifamily 2006 2007 2008 Base Package MF $250 $250 $250 Exterior Lighting $20 $20 $20 Interior Lighting $20 $20 $20 Lighting 10 E CLFs NA NA $40 Cooperative Marketing As an additional incentive to builders, a cooperative marketing opportunity is available to builders who wish to utilize the ENERGY STAR logo and marketing resources. Builders who develop ENERGY STAR approved marketing materials are eligible to receive marketing funds of up to one-third of the total marketing cost. Available marketing funds range from $2,500 to $10,000, depending upon the number of homes constructed. Program Assumptions and Participation The method by which energy savings are attributed to the program has also evolved within the evaluation period. Prior to mid-2008, deemed savings values for single-family and multifamily construction were estimated using building simulations modeled to each tier level. A weighted average of these savings was calculated for each home type single-family or multifamily then attributed to all homes of that construction type, regardless of tier level or addition of plus measures. Beginning in June 2008, program savings estimates were expanded to include annual savings estimates attributable to each tier level as well as for the individual plus measures. Table 10 presents reported participation for the 2006 2008 program period as well as savings estimates through May 2008. A breakout of savings estimates used for homes incented through the program, beginning in June 2008, is provided in Table 10. The Cadmus Group, Inc. / Energy Services 10
Table 9. 2006-2008 Reported 4 Assumptions and Participation Savings Per Home Number of Homes Total Savings (kwh/year) Total Program Year SF MF SF MF SF MF Savings 2006 1,931 915 1,758 0 3,394,698 0 3,394,698 2007 1,623 681 2,184 88 3,544,632 59,928 3,604,560 2008 Before June 1,298 545 1,065 65 1,382,796 35,412 1,418,208 2008 - After June 1,691 473 547 167 925,241 79,048 1,004,289 Total NA NA 5,554 320 9,247,367 174,388 9,421,755 Table 10. June December 2008 Reported Assumptions and Participation Home Savings - Post-June 2008 Savings Per Home (kwh/year) Number of Homes Total Savings Tier 1 1,198 355 425,432 Tier 2 1,594 182 290,181 Tier 3 1,997 1 1,997 Tier 4 2,402 8 19,219 Multifamily 320 167 53,440 GSHP Option 12,454 1 12,454 Home Total NA 714 802,723 Plus Measure Savings Post-June 2008 Savings Per Measure Number of Homes Total Savings 14 SEER HVAC 158 16 2,534 Duct Placement 486 277 134,511 ENERGY STAR Ceiling Fan 68 2 136 High-Efficiency Evaporative Cooler 736 1 736 Whole House Fan System 288 3 864 ENERGY STAR Dishwasher 24 526 12,624 Lighting: 15 additional CFLs 396 62 24,552 Lighting: MF 10 additional CFLs 264 97 25,608 Measure Total 201,566 June December 2008 Total 1,004,289 As shown in Table 12, the Tier 1 home type was the most commonly constructed home option incented through the program. 4 Savings and participation as reported in DSM goals and year-end reports. Incented home counts reported in 2006 and 2007 differ from those found in the participant database provided by the implementer. There are approximately 60 additional homes in the participant database; the implementer was unable to identify a reason for the difference. Savings are adjusted by 80% NTG for consistency with 2006-2007. The Cadmus Group, Inc. / Energy Services 11
Table 11. Home Types Incented during 2006 2008 Evaluation Period Single-Family 2006 2007 2008 Home Total Tier 1 944 2,016 1,385 4,345 Tier 2 856 158 217 1,231 Tier 3 NA NA 1 1 Tier 4 NA NA 8 8 GSHP Option NA NA 1 Total Single-Family Homes 1,800 2,174 1,612 5,586 Multifamily 2006 2007 2008 Home Total Base Package (Total Multifamily Homes) NA 116 232 348 Total Participant Homes 1,800 2,290 1,844 5,934 Of Plus measure options, the ENERGY STAR dishwasher was the most commonly incented, as shown in Table 13. Table 12. Plus Measures Incented during 2006 2008 Evaluation Period Single-Family 2006 2007 2008 Measure Total GSHP Option NA NA 1 1 14 SEER HVAC NA NA 16 16 ENERGY STAR Dishwasher 1,149 1,360 1,269 3,778 Duct Placement NA NA 277 277 Duct within Conditioned Space 19 880 516 1,415 Ducted Evaporative Cooling System NA NA 1 1 ENERGY STAR ceiling fan NA NA 4 4 Lighting 1 288 124 413 Whole House Fan System NA NA 3 3 SF Total 1,169 2,528 2,211 5,908 Multifamily 2006 2007 2008 Measure Total Plus Measures NA 81 172 253 MF Total NA 81 172 253 Program Total 1,169 2,609 2,383 6,161 Program Logic and Progress Indicators The ENERGY STAR New Homes Program seeks to develop strategies and activities to mitigate key market barriers to adoption of efficient construction practices and inclusion of energyefficient equipment in new construction. A summary of these barriers, strategies, and activities, and short-term and long-term outcomes (progress indicators) by market actor is presented in Figure 1. As demonstrated in the diagram, the program uses training and education to complement availability of multiple incentive types. The Cadmus Group, Inc. / Energy Services 12
Figure 1. Program Logic Model and Progress Indicators for the ENERGY STAR New Homes Program The Cadmus Group, Inc. / Energy Services 13
3. Impact Evaluation Methodology This analysis was designed to determine program savings and cost-effectiveness, and included a billing analysis, interviews with dealers and distributors, and customer surveys. Each of these is discussed below. Billing Analysis Home Sample Selection and Data Collection The implementer provided program tracking and characteristics data for each participant home incented during the evaluation period. The measure data included measures incented through the program, home address, occupant name and contact information, floor area, housing type (singleor multifamily), number of floors, and space and water heating energy sources. Billing data were provided by Rocky Mountain Power for all new residential hookups, beginning in 2005 and ending in November 2009. Many of the characteristics listed above, including square feet, were available for most new residential hookups. We determined the date of occupancy for all the new hookups using information from the participant data set to identify participant homes from among the population of new hookups. Once these participating residential homes were identified, remaining homes were assigned to the nonparticipating sample frame. Billing Data Weather Matching and Data Screening The zip code associated with each new-residential hookup provided by Rocky Mountain Power was used to map the home to the nearest weather station. We then obtained historical weather data from 2005 through 2009 for all associated Utah weather stations. From the average daily temperature, we obtained base 65 heating degree days (HDD) and cooling degree days (CDD). For each billing period record, we matched the associated HDDs and CDDs. As part of the screening process, nonresidential and non-single-family homes were removed. The program participation date was used to assign the program year to the participants. However, in many cases, billing data associated with these participant homes were initially under builder account names. To ensure we used only customer billing data, we selected billing data only after the first occupant moved into the home. Next, we examined each year of billing data after the first installation. Any year with fewer than nine months (270 days) of billing data was removed from analysis because they would not provide sufficient data to run the PRISM equivalent modeling approach. Sample Matching Since the billing analysis relied only on the post-installation data, it was important for the participant and nonparticipant groups to be matched on characteristics, particularly home square footage. Based on program data, participant homes tended to be larger than nonparticipant homes. The Cadmus Group, Inc. / Energy Services 15
Square footage from participant program tracking data was used for participants, wherever available. When these were missing, Rocky Mountain Power square footage estimates available for both participant and nonparticipant homes were used. Some of the homes over 5,000 square feet were compared to a real estate database for reliability. While these square footage estimates were not necessarily reliable for each individual home, the average was within 5% of the real estate-based square footage. Moreover, because of the importance of normalizing square feet, the real estate database was also used to estimate square footage for approximately 4% of the homes that did not have data available in the program tracking data or in Rocky Mountain Power customer data. Additionally, approximately 5% of the homes were excluded from the analysis due to lack of square footage data. All homes with square footage estimates were included in the analysis. Figure 2 compares square footage of homes used in this analysis by occupied year. From this figure, it is clear participants generally had larger homes in 2006 and 2007. In 2008, home sizes dropped significantly for both participant and nonparticipants. Rocky Mountain Power noted that the square footage of 2009 homes appear to be in line with 2006 and 2007 program homes. Figure 2. Participant and Nonparticipant Square Footage by Program Year Energy Analysis PRISM-like Modeling For each participant and nonparticipant home, a PRISM-like heating and cooling model was estimated for each year after the initial year of construction to weather-normalize raw billing data usage. The model specification used was: For each customer i and month t, ADC it= α i + β 1 AVGCDD it + β 2 AVGHDD it + ε α i is the intercept for the participant (or nonparticipant); this represents the base load it The Cadmus Group, Inc. / Energy Services 16
β 1 β 2 ADC is the model cooling slope is the model space heating slope it AVGCDD AVGHDD ε it is average daily consumption during the program period it it is the error term, is average daily CDDs for the specific location is average daily HDDs for the specific location From the model above, the weather-normalized annual consumption (NAC) was computed as follows: Where, for each customer i, NAC i = α i * 365 + β 1 * LRCDD i + β 2 * LRHDD i + ε αi is the intercept that is the average daily or base load for each participant (or nonparticipant); this represents the average daily base load from the model NACi is the normalized annual consumption LRCDDi, is the annual, long-run CDDs based on home location LRHDDi is the annual, long run HDDs based on home location αi * 365 is the annual base load usage εit is the error term In addition, a heating-only model (AVGHDD term only) and a cooling-only model (AVGCDD term only) were estimated, and models with negative parameters were excluded. From these best models, we were able to estimate NAC. Since some program years had more than one year of valid models (for example, 2007 participants had potentially 2007, 2008, 2009 NACs), we averaged weather normalized usage across the various years of occupancy. The yearly, weathernormalized usages were not expected to vary significantly by year within a home; this ensured each participant and nonparticipant had similar weighting, regardless of program year. Usage Per Square Foot Distribution As the square feet shown in Figure 2 indicated differences in sizes of participant and nonparticipant homes, all analyses for determining program savings were developed as kwh per square foot (kwh/sqft). The kwh/sqft distributions for participants and nonparticipants were compared in detail. Figure 3 shows the distribution of kwh/sqft by percentile for the 2006 2008 participants and nonparticipants. From these, it became clear normalized usage per square foot for nonparticipants was larger than participant usage. However, the chart reveals some kwh/sqft estimates were too low. This may be due to customers purchasing the home but not yet occupying it or simply an unreliable square footage estimate. To remove these unreasonably low readings, homes with less than 1.5 kwh/sqft were assumed vacant or unoccupied, and were dropped from the analysis. it The Cadmus Group, Inc. / Energy Services 17
Figure 3. Usage Per Square Foot Distribution (Participant and Nonparticipant Comparison) Nonparticipants Participants Removed < 1.5 kwh/sqft (5% of homes) 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5 11 11.5 12 KWH_SQFT Figure 4 summarizes usage distribution once this screen was placed on the data. The final analysis uses this subgroup of non-vacant, customer homes. The Cadmus Group, Inc. / Energy Services 18
Figure 4. Usage Per Square Foot Distribution (Without Vacant/Unoccupied Homes) Nonparticipants Participants 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5 10.5 11.5 KWH_SQFT Usage Per Square Foot Savings Estimation Model To obtain yearly savings estimates, a model was developed using the entire set of 2006 2008 homes passing the screening procedure above. The following final model specification was used to derive the kwh per year savings is as follows: NAC_SQFT i= β 1 Y06 i + β 2 Y07 i + β 3 Y08 i + β 4 Y06 i * PART i + β 5 Y07 i * PART i + β 6 Y08 i * PART i + β 7 SQFT08 i + ε For each customer i NAC_SQFTit is the average normalized annual kwh per square foot for customer i Y06i is a dummy variable that is 1 for 2006 homes, and 0 otherwise. Y07i is a dummy variable that is 1 for 2007 homes, and 0 otherwise. Y08i is a dummy variable that is 1 for 2008 homes, and 0 otherwise. PART I is a dummy variable that is 1 for participant homes, and 0 otherwise. SQFT08I s a dummy variable that is 1 for participant homes, and 0 otherwise. β1 is the 2006 nonparticipant usage per square feet β2 is the 2007 nonparticipant usage per square feet β3 is the 2008 nonparticipant usage per square feet it The Cadmus Group, Inc. / Energy Services 19
β4 is the 2006 participant savings per square feet β5 is the 2007 participant savings per square feet β6 is the 2008 participant savings per square feet β7 is the additional usage per square foot in 2008 εit is the error term The SQFT08 was included in the savings modeling to account for significantly lower average square footages in 2008, as seen in Figure 1. This allowed us to obtain comparable and more robust savings estimates for each year. The kwh per square feet savings estimates for the 2006 to 2008 years were obtained directly from the β 4, β 5, β 6 coefficients from the model specification above. Freeridership and Net-to-Gross The percent of savings that would have occurred in the program s absence freeridership was calculated through surveys with program participants. In calculating freeridership, Cadmus conducted telephone surveys with 20 builders, representing 19 companies responsible for 1,185 homes incented through the ESNH program in years 2006 through 2008. Participant builders were selected randomly from among the population of builders who received incentives through the ESNH program. Given the relatively small number of builders surveyed, a single freeridership value was calculated for the three-year evaluation period. This value was used to determine the Net-to-Gross (NTG) ratio for the program. Spillover Spillover occurs when customers choose to purchase energy-efficient measures or adopt energyefficient building practices as a result of a program, but without directly participating in a program. As a qualitative assessment of the potential for spillover among participant builders, we asked if additional energy-efficiency technologies had been installed in any of their other new homes since they began participating in the Rocky Mountain Power program, but for which they received no incentive. If they answered affirmatively, we then asked, How influential was the program in your decision to install these additional measures? Air Conditioner Site Visits At the request of Rocky Mountain Power, Cadmus conducted site visits to 13 incented homes in conjunction with site visits that were being conducted in close proximity for another Rocky Mountain Power program evaluation. The sample was not statistically derived but served to explore the correct sizing of air conditioning (AC) units being installed through the program. The results of the billing analysis, freeridership, spillover, and air conditioner site visits are presented within the Impact Evaluation Results chapter. Cost-effectiveness results are presented in a separate Cost-Effectiveness chapter. The Cadmus Group, Inc. / Energy Services 20
4. Process Evaluation Methodology The process evaluation consists of a materials review and a review of best practices, interviews with program staff and market actors, and surveys with participant and nonparticipant homeowners. Materials and Best Practices Review To help understand Rocky Mountain Power s ENERGY STAR New Homes program offerings, as well as to structure and guide discussions, it was important to gain a comprehensive understanding of program objectives, activities comprising the program effort, and its accomplishments to date. To do so, Cadmus reviewed the following relevant documents: Rocky Mountain Power s ENERGY STAR New Homes Web site. ENERGY STAR for New Homes Sponsor and Utility Partner Guide. Builder and HERS rater application and inspection documents. Cooperative marketing materials, including sample materials and the Selling ENERGY STAR New Homes presentation. Implementer Scope of Work, 2004. Monthly reports submitted by Ecos. In addition, secondary research into industry best practices was conducted, with a focus on the National Energy Efficiency Best Practices Study: Volume R8 Residential New Construction Best Practices Report, submitted to the California Best Practices Project Advisory Committee in December 2004. 5 In addition to providing context for this evaluation, the research into best practices enabled a comparison to be made between Rocky Mountain Power s ENERGY STAR New Homes program and best practices benchmarks developed in the 2004 study. Interview and Survey Data Collection Interviews and surveys were used as the primary means of conducting the process evaluation. Table 14 presents the interview and survey activities conducted for this process evaluation. 5 Quantum Consulting, Inc. December 2004. The Cadmus Group, Inc. / Energy Services 21
Rocky Mountain Power Program Staff Table 13. Interview/Survey Data Collection Implementer Staff and HERS Raters ENERGY STAR Home Buyers Participating Nonparticipating Survey Area Builders Builders Number of respondents 3 3 20 20 100 100 Program features Incentive levels Program achievement Participant satisfaction Marketing awareness Conventional Home Buyers Utility and Staff Interviews Cadmus staff interviewed multiple parties to gain an in-depth understanding of Rocky Mountain Power and its ES New Homes program, including key members of utility staff, implementation staff directly involved in the operations of the program, and HERS raters. Because ES New Homes is implemented by a third-party, utility staff interviews primarily focused on high-level management and regulatory issues, while program-level details were generally addressed in interviews with implementers and raters. Results of these interviews, presented in the Process Evaluation Interview Results chapter, reflect this division of focus, with portfolio management and regulatory issues presented in the Utility Program Management section, and program level details contained in the ES New Homes Implementation and Field Work section. Builder and Homeowner Surveys Surveys were conducted to gain information from participant and nonparticipant builders and homeowners. Specific topics of focus and descriptions of sample selection technique and methodologies are provided below. Results of these activities are presented in the Process Evaluation Survey Results chapter. Participant and Nonparticipant Builders Cadmus conducted telephone surveys with 21 builders participating in the ENERGY STAR New Homes program between 2006 and 2008, as well as 20 nonparticipant builders who had constructed new homes during the same time period, but without the benefit of an incentive. Questions asked of the participant group focused on the following subject areas: Program participation and awareness Coordination with Questar Application process Home energy raters Building practices and freeridership Building codes Marketing Program satisfaction The Cadmus Group, Inc. / Energy Services 22
Nonparticipant surveys focused on the following: Program awareness and participation decisions Energy-efficient building practices Current building practices Sample Selection and Methodology Participant Builders. Builders constructing homes through the program were randomly selected from a contact list provided by the implementer. These builders were offered a $50 gift card to Home Depot, Lowe s, or Gastronomy restaurants as an incentive to complete the survey. A Web survey was also made available to builders requesting the survey be e-mailed. Nonparticipant Builders. Builders constructing homes in Utah during the evaluation period, but who had not received any incentives, were also offered an incentive to complete a phone or Web-based survey. A sample frame was generated by identifying home builder association members identified as builders, and cross-referencing this list with the list of participants provided by the implementer. Builders identified from the participant list were removed, and builders from the resulting nonparticipant list were called randomly for the survey. Participant and Nonparticipant Homeowners To gauge homeowner awareness of the program and ENERGY STAR building practices, surveys were conducted with homeowners who purchased new homes during the program evaluation period. Participant homeowner surveys focused on the following areas: Program awareness Influence of industry workers Decision to participate Participation in other utility programs Energy expectations Nonparticipant homeowner surveys were designed with similar focus areas. Sample Selection and Methodology Participant Homeowners. Homes which had been incented through the program during the evaluation period were randomly selected from a contact list provided by the implementer. A third-party survey firm was used to conduct calls to these homeowners. A total of 101 surveys were completed among this group. Nonparticipant Homeowners. Homes constructed within Rocky Mountain Power service territory during the evaluation period, but not receiving any incentives, were also surveyed. A sample frame was generated by cross-referencing the list of participating homes with a utilityprovided list of all new meter hookups during the evaluation period. Homes identified from the participant list were removed, and remaining homeowners were called randomly for the survey. A total of 102 surveys were completed among this group. The Cadmus Group, Inc. / Energy Services 23
5. Impact Evaluation Results To estimate program gross energy savings, we analyzed billing data for program homes and nonparticipant homes, and compared the weather-adjusted annual energy consumption to derive gross savings. As described in the Impact Methodology section, we used a PRISM-like methodology that explicitly includes actual weather data represented by HDDs and CDDs provided for the nearest weather station. An analysis of freeridership and spillover was also conducted, as was verification of sizing practices. Impacts of this research are presented below, including program energy savings, freeridership, and spillover results. Gross Energy Savings Estimates Table 15 summarizes the square feet, expected savings, model-based savings, and the precision level. The lower savings estimates for the 2008 program year primarily appear to relate to construction of ENERGY STAR homes smaller than average, compared to the other years. Moreover, 2008 expected savings per square foot were the highest in 2008 across the various groups. Table 14. Expected Savings and Model Based Savings and Realization Rates 2006 2007 2008 Total Single-Family Homes 1,800 2,174 1,612 Single-Family Homes (Model N) 896 1,119 1,466 Nonparticipants (Model N) 15,968 11,010 9,989 Expected Savings per Participant 1,931 1,623 1,431 Average Square Feet (Participants) 2,874 2,898 2,318 Expected Savings Per Square Foot 0.67 0.56 0.77 Savings Per Square Foot (Model) 0.69 0.52 6 0.52 Savings KWh (Model) 1,985 1,517 1,213 Savings RR (Model) 103% 93% 85% Precision of Estimate (Model) 19% 23% 24% Table 16 presents gross evaluated savings by home type for each program year as well as savings reported to the utility by the implementer. Multifamily savings estimates were calculated by applying the realization rate estimated for single-family homes to reported savings for multifamily homes in each program year. As mentioned above, the lower realization rate in 2008 appears to relate to the smaller construction of ENERGY STAR homes. 6 A code change occurred in January 2007 when the 2006 IECC Residential Code was adopted. The Cadmus Group, Inc. / Energy Services 25
Table 15. ESNH Program Gross Savings, 2006 2008 7 Savings Type Home Type 2006 2007 2008 Total Single-Family 3,394,698 3,544,632 2,305,520 9,244,850 Reported Savings Multifamily 0 59,928 116,977 176,905 (kwh) Total 3,394,698 3,604,560 2,422,497 9,421,755 Single-Family 3,573,078 3,298,560 1,955,483 8,827,121 Evaluated Multifamily 0 56,024 99,156 155,180 Savings (kwh) Total 3,573,078 3,354,584 2,054,639 8,982,301 Realization Rate 105% 93% 85% 95% Precision of Estimate 19% 23% 24% 13% Freeridership Results The percent of savings that would have occurred in the program s absence freeridership was calculated through surveys with program participants. In calculating freeridership, Cadmus conducted telephone surveys with 20 builders representing 19 companies responsible for 1,185 homes through the ESNH program in years 2006 through 2008. Participant builders were selected randomly from among the population of builders receiving incentives through the ESNH program. Given the relatively small number of builders surveyed, a single freeridership value was calculated for the three-year evaluation period. This freeridership value was used to determine the NTG ratio for the program. As presented in Table 17, the NTG value was applied to the total gross evaluated savings, resulting in total net evaluated savings for this program. Table 16. ESNH Program Freeridership Program Year Freeridership NTG Ratio Gross Savings Net Savings 2006 3,573,078 2,658,370 2007 3,354,584 2,495,811 26% 74% 2008 2,054,639 1,528,651 Total All Years 8,982,301 6,682,832 Spillover Results As a qualitative assessment of the spillover potential among participant builders, we asked if additional energy-efficiency technologies had been installed in any of their other new homes since they began participating in the Rocky Mountain Power program, but for which they received no incentives. If they answered affirmatively, we asked: How influential was the program in your decision to install these additional measures? As Table 18 shows, of 21 responses to the first question, 10 indicated additional energyefficiency measures or technologies had been installed without direct benefit from the program. The Cadmus Group, Inc. / Energy Services 26
Table 17. ESNH Program Spillover (n=21) Question Yes No Don t Know Installed additional energy-efficiency measures/technologies since 10 (48%) 6 (29%) 5 (24%) participating in the program, but without benefit of an incentive? Additional measures and technologies identified by builders providing examples included: Zero Net Energy construction Insulation Furnaces When asked to rate the program s influence level on the decision to install these additional measures or technologies, six of the 10 responses indicated the program was Influential or Very Influential. The surveys show the program had an additional spillover effect on new home construction, but we did not attempt to quantify this impact. Air Conditioner Site Visit Results The non-statistical sample of site visits conducted at the request of Rocky Mountain Power indicated that an issue surrounding the correct sizing of air-conditioning units incented through the program may exist. As a result of these site visits, Rocky Mountain Power commissioned a more rigorous study (the Ecos Study) to explore this issue in greater detail. The Ecos study, conducted by Ecos Consulting in partnership with Ecotope, Inc. and Delta T, found that 55% of the air-conditioners installed between 2006 and 2010 in the sampled homes were oversized, with 27% of these oversized by more than ½ ton 8. The Ecos Study reported that 31% of the airconditioners installed in the sample homes were correctly sized, and that 14% were undersized. As a result of the Ecos Study, Ecos developed a set of recommendations 9 for improving the implementation/verification process. Key Findings The billing analysis demonstrates the ENERGY STAR New Homes program reduced energy consumption of homes incented through the program. However, the level freeridership had negative impact on energy savings. Additionally, the construction of smaller homes in 2008 may have resulted in a lower realization rate for that year. 8 The Ecos Study, pg. 16 9 The Ecos Study, pg. 21 The Cadmus Group, Inc. / Energy Services 27