Connexus Energy Ramsey, MN Measurement and Verification Report of OPower Energy Efficiency Pilot Program July 28, 2010 Contact: Chris Ivanov 1532 W. Broadway Madison, WI 53713 Direct: 608-268-3516 Fax: 608-222-9378 Email: ivanovc@powersystem.org Web Site: www.powersystem.org Madison, WI. Minneapolis, MN. Marietta, OH. Indianapolis, IN. Sioux Falls, SD
Confidential, Copyrighted, and Proprietary This document contains information confidential to Connexus Energy (Connexus or Cooperative) and Power System Engineering, Inc. (PSE). Unauthorized reproduction or dissemination of this confidential information is strictly prohibited. Copyright 2010 Power System Engineering, Inc. This document includes methods, designs, and specifications that are proprietary to Power System Engineering, Inc. Reproduction or use of any proprietary methods, designs, or specifications in whole or in part is strictly prohibited without the prior written approval of Power System Engineering, Inc. NEITHER POWER SYSTEM ENGINEERING, INC. NOR CONNEXUS ENERGY SHALL BE RESPONSIBLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES (INCLUDING LEGAL FEES AND COURT COSTS) ARISING OUT OF OR CONNECTED IN ANY WAY TO THE UNAUTHORIZED USE, MODIFICATION, OR APPLICATION OF THIS DOCUMENT OR THE PROPRIETARY INFORMATION, METHODS, AND SPECIFICATIONS SET FORTH IN THIS DOCUMENT, WHETHER IN WHOLE OR IN PART.
TABLE OF CONTENTS 1.0 Executive Summary... 1 2.0 Background of the Program... 4 3.0 Impact Evaluation Approaches and Results... 6 3.1 True Impact Test Approach and Results... 6 3.2 OLS Model Approach and Results... 8 3.3 Fixed Effects Model Approach and Results... 10 4.0 Conclusion... 13 5.0 Appendix... 14 5.1 Thoughts on Further Research... 14 5.2 Model Results... 15 5.3 PSE Background... 16 5.4 Authors... 16
1.0 Executive Summary
1.0 1.0 Executive Summary In response to Minnesota s new state-wide conservation goals, Connexus Energy (Connexus) partnered with OPower, formerly Positive Energy, to launch a new energy efficiency program. Power System Engineering, Inc. (PSE) was hired as an independent third-party evaluator for this pilot program. PSE s role was to validate the methods and estimates of energy savings attributed to the program in the first year. Data was provided to PSE by OPower staff; in particular, Tyler Curtis. This included monthly billing data, program-specific characteristics, customer demographics, and climatic data. The time span of the data stretched from January 2007 to January 2010. This time period encompassed over two years of data before the pilot began (January 2007 through February 2009) and 11 months of data during the pilot (March 2009 through January 2010). The dataset included over 2.5 million observations encompassing nearly 80,000 member accounts. PSE examined the sample selection, program design, and data outputs. In evaluating the energy savings of the program, three measurements were calculated. These included the True Impact Test, an Ordinary Least Squares econometric model, and a Fixed Effects econometric model. These evaluation techniques are standard methods of measuring energy efficiency impacts. The True Impact Test is a non-parametric analysis which examines the change in the differences of the control and treatment groups from pre-pilot to post-pilot time periods. The advantage of this method is that it is relatively simple to understand and is a powerful tool in evaluating program impacts. In algebraic terms, the True Impact Test can be described by the following equation: The other two evaluation methods use econometric analysis to estimate a specified model. Each model calculates a parameter estimate of the impacts of the pilot program on energy usage. The first model uses an Ordinary Least Squares (OLS) estimation procedure. The second model Connexus OPower Energy Measurement-Verification Report 1 MN0480806 8/9/10
1.0 employs a Fixed Effect estimation procedure, which is the same approach that OPower uses in its models. The advantage of the OLS model is that it allows the researcher to include customer-specific information (e.g., income, housing structure, age) into the regression. This enables estimation of demographic impacts on electricity usage. The advantage of the Fixed Effects model is that it automatically adjusts for all participant differences in estimating the program impacts. The downside of the Fixed Effect model is that it cannot include time invariant customer information. For example, many demographic variables such as sex or race do not vary over time and cannot be included in the Fixed Effects Model. The three estimates are presented in the table below. There is stability in results across the three estimation procedures. The True Impact Test estimates a daily kilowatt hour (kwh) reduction of 0.621 for each participating member of the pilot program during the post-pilot time period. Similarly, reductions of 0.622 and 0.637 are seen for the OLS and Fixed Effects models, respectively. Reduction percentages range from 2.05 to 2.10 percent. Estimated Per Customer Savings of OPower Program Annual KWH Savings per customer Daily Annual Percentage Reduction True Impact Test 0.621 227 2.05% OLS Model 0.622 227 2.05% Fixed Effects Model 0.637 232 2.10% Average: 0.626 229 2.07% Given the stability of the results derived from separate estimation procedures, the large sample size, and the randomness of sample selection, PSE concludes there are tangible energy savings resulting from this program. Results are robust with a high degree of confidence attached to them. Connexus OPower Energy Measurement-Verification Report 2 MN0480806 8/9/10
1.0 OPower s fixed effects model estimated a program impact of 2.27%. This is slightly higher than PSE s Fixed-Effect model which estimated a reduction of 2.10%. The small disparity in results is most likely due to the OPower process of continually updating their results as new data becomes available and using a rolling program start date commencing after the second Home Energy Report mailing was received by a particular household. Further research is required to better quantify the long-run impacts of this program. Research items, such as estimating the impacts of multiple years of treatment and the possible legacy impacts of treatment after the program is discontinued, would have important ramifications on cost benefit tests and on optimal program design and cycling. Research on customer demographics and their impact on program energy savings would also add value. Additionally, interaction effects with other energy efficiency programs would be of interest. Along these same lines, determining the ways the pilot participants lowered energy usage would be important information (e.g., conservation efforts, appliance replacement). Connexus OPower Energy Measurement-Verification Report 3 MN0480806 8/9/10
2.0 Background of the Program
2.0 2.0 Background of the Program In response to Minnesota s new state-wide conservation goals, Connexus Energy (Connexus) partnered with OPower to launch a new energy efficiency program. Power System Engineering, Inc. (PSE) was hired as an independent third-party evaluator for this pilot program. PSE s role was to validate the methods and estimates of energy savings attributed to this program. The program provides residential participants with a Home Energy Report designed to motivate and educate recipients to improve the energy efficiency of their homes. The program is designed as a large scale behavioral experiment. Control and treatment groups were randomly selected out of a total of 80,000 households. Each group consisted of approximately 40,000 households. Households had to meet the following criteria to be included in one of the groups: 1. Exactly one active electric account per household. 2. Account history dating to January 2007. 3. Valid meter read cycle. 4. Not on a medical rate plan. 5. At least 50 neighbors. The Home Energy Report relies on providing residences with a comparison of energy usage to their neighbors, thus the definition of what constitutes a neighbor is important. OPower defines the neighbor benchmark by the following characteristics. Size of home: Neighbors are selected based on similarity in home size. Homes greater or less than 25 percent of the participant s home are excluded from the comparison. Distance: A radius of 0.5 miles of the targeted household is searched in order to find 100 similar households. At least 50 households must be available within this radius for it to be included in the pilot. Heating Fuel: Households are compared with other homes that have the same heating fuel. Connexus OPower Energy Measurement-Verification Report 4 MN0480806 8/9/10
3.0 Impact Evaluation Approaches and Results
3.0 3.0 Impact Evaluation Approaches and Results Section 3.0 provides a more detailed look at the three approaches used to quantify electrical savings resulting from the pilot program. Research results and findings are provided for each approach. These three approaches do not depend on each other. Thus, they are independent estimations with only the data as a common element. The three approaches all reveal similar findings. The pilot program did induce consumers to lower energy use, provoking annual savings of about 229 kwh per participant. This amounts to around 9,160 MWh annually saved by the 40,000 participants. The True Impact Test estimated savings of about 2.05 percent, the OLS Model estimated savings at 2.05 percent, and the Fixed Effects Model estimated savings of 2.10 percent. Results for the OLS and Fixed Effects Models were statistically significant at a 99 percent confidence threshold. 1 Estimated Per Customer Savings of OPower Program Annual KWH Savings per customer Daily Annual Percentage Reduction True Impact Test 0.621 227 2.05% OLS Model 0.622 227 2.05% Fixed Effects Model 0.637 232 2.10% Average: 0.626 229 2.07% 3.1 True Impact Test Approach and Results The True Impact Test is the most straightforward method of evaluating energy efficiency programs. The True Impact Test is a non-parametric analysis which examines the change in the differences of the control and treatment groups from pre-pilot to post-pilot time periods. The advantage of this method is that it is relatively simple to understand and examine the results and is a powerful tool in evaluating program impacts. In algebraic terms, the True Impact Test can be described by the following equation: 1 The True Impact Test is a non-parametric approach to estimating savings and thus is not able to provide a confidence level based on statistical tests. Connexus OPower Energy Measurement-Verification Report 6 MN0480806 8/9/10
3.0 To calculate the True Impact Test, PSE took the difference in the average daily use of the treatment group during the pilot ( ) to the average daily use of the control group during the pilot ( ) and then subtracted this difference from the difference in the average daily use of the treatment group before the pilot started ( ) to the average daily use of the control group before the pilot began ( ). This provides an estimate of the change in the post-pilot treatment energy use that can be attributed to the pilot itself. Given the large sample size and design of the program, this estimate is convincing. The graph below further illustrates the finding. The difference between the treatment and control group significantly widens after the start of the energy efficiency program. 30.500 30.000 29.500 29.000 28.500 28.000 27.500 27.000 26.500 Daily KWH Use per Customer of Test Groups 30.288 30.222 Pre-Treatment Period Control Group 28.622 Treatment Group 27.935 Treatment Period In an attempt to separate the seasonal differences in the impacts, the seasons were divided into three categories. 1. Winter (December, January, February, March, April). 2. Summer (July, August, September). 3. Shoulder (May, June, October, November). Connexus OPower Energy Measurement-Verification Report 7 MN0480806 8/9/10
The table below shows the average daily use per group on an annual, winter, summer, and shoulder basis. Treatment energy savings in the summer are estimated at 0.647 kwh per day. This is a reduction of 1.82 percent over the control group s usage. The winter months show an impact of 0.677 average kwh reductions per day. The shoulder months have an estimated reduction in average daily use of 0.517 kwh. True Impact Test of OPower Program (Daily KWH Use per Customer) Annual Winter Summer Shoulder Pre-Treatment Period (Jan. 2007-Feb. 2009) Control Group 30.288 30.779 35.639 25.473 Treatment Group 30.222 30.760 35.497 25.392 Difference (T-C) -0.066-0.019-0.142-0.081 Treatment Period (March 2009 - January 2010) Control Group 28.622 29.557 31.184 25.697 Treatment Group 27.935 28.861 30.395 25.100 Difference (T-C) -0.687-0.696-0.789-0.598 3.0 Treatment KWH Impact: -0.621-0.677-0.647-0.517 Percent Reduction: -2.05% -2.20% -1.82% -2.03% 3.2 OLS Model Approach and Results To further substantiate the True Impact Test results, an Ordinary Least Squares regression was estimated. This regression incorporated pre-pilot data (January 2007 to February 2009) and the post-pilot data March 2009 to January 2010) and estimated variables to determine their impact on average daily electricity use. Six explanatory variables were included in the OLS Model. 1. Intercept Term (a): This variable measures the expected electricity usage with zero values of the other variables considered. This would be the expected usage if there was no pilot, zero heating degree days, and zero cooling degree days. 2. Test Indicator Binary Variable (T): This variable equals 1 if the customer was selected to be in the treatment group and 0 if the customer was placed in the control Connexus OPower Energy Measurement-Verification Report 8 MN0480806 8/9/10
group. These values are constant across the pre-pilot and post-pilot time frames. This variable allows the researcher to estimate the inherent differences between the two groups. 3. Post-Pilot Binary Variable (P): This variable equals 1 if the time period is after the start of the pilot program (March 2009). It equals 0 if the time period is before the start of the pilot program (prior to March 2009). This variable allows an estimation of the impact on average daily use of the different time periods. 4. Test Indicator multiplied by the Post-Pilot (P*T): This variable equals 1 if the observation is in the treatment group and the month is after the start of the pilot. It equals 0 if the observation is either in the control group or is in the treatment group but before the pilot start date. This variable estimates the impact the pilot program has on average daily use. PSE s estimate of program savings is taken from the estimated coefficient on this term. 5. Heating Degree Days (HDD): This variable measures the heating degree days for the billing month. The parameter estimates the impact that heating degree days have on average daily usage. 6. Cooling Degree Days (CDD): This variable measures the cooling degree days for the billing month. The parameter estimates the impact that cooling degree days have on average daily usage. The estimated equation takes the following functional form: 3.0 Econometric analysis is used to estimate the values of a, b, c, d, e, f in order to minimize the squared sum of the error term ( ). The parameter estimates (b,c,d,e,f) are interpreted as the marginal energy use of the variable they are multiplied by. For example, parameter f is a measure of how the average daily use will increase when cooling degree days increase by one. In the context of this report, the most interesting parameter is d. This measures the marginal impact on average daily use of being in the treatment group during the pilot period. The parameter estimate for d is -0.622, meaning that average daily use is estimated to be reduced by over half a kwh if the participant is currently being sent Home Energy Reports. Connexus OPower Energy Measurement-Verification Report 9 MN0480806 8/9/10
3.0 Estimated Impacts OLS Model Daily kwh -0.622 Percent Reduction -2.05% While the econometric method is more complicated and less easy to comprehend relative to the True Impact Test, it does have the advantage of being able to test how much confidence policy makers can have on the estimated impact. One way to do this is to calculate the t-statistic on the parameter estimate d. PSE estimates a t-statistic of 12.40. For context, a t-statistic of 1.645 indicates a 90 percent level of confidence and a t-statistic of 2.326 indicates a 99 percent degree of confidence. A t-statistic of 12.40 is, therefore, highly significant and offers a great deal of confidence in the result. 3.3 Fixed Effects Model Approach and Results To further substantiate the results of the pilot, PSE estimated a Fixed Effects Model. A Fixed Effects Model is an econometric method used to capture all household-specific effects on energy consumption. It is quite unlikely that household characteristics would significantly influence the estimated impact of the program, given the large sample size and random selection of the treatment and control groups. However, a Fixed Effect Model can alleviate any potential concerns. The logic behind the approach is to estimate separate intercepts for each customer. These intercepts incorporate the household-specific characteristics that influence energy consumption. It is implausible to actually estimate 80,000 separate intercepts, as is the case for this dataset. However, econometricians have discovered a computational trick which leads them to the same estimation results. By subtracting all included variables by the average of the variable for each individual household, the same parameter estimates are attained. The Fixed Effects Model has three main disadvantages. 1. Degrees of freedom loss. By implicitly including 80,000 intercept terms, the estimation loses 79,999 degrees of freedom. The loss of degrees of freedom makes the Connexus OPower Energy Measurement-Verification Report 10 MN0480806 8/9/10
estimation less precise. However, given that the dataset is so large and contains over 2.5 million observations, this loss is negligible. 2. Variables that do not vary with time. The transformation involved in the Fixed Effects Model wipes out the possibility of including variables that do not vary with time. This limits the researcher s ability to identify variables which influence program impacts to only those variables where data is available and varies over time for each household. For example, an income variable that does not vary could not be included in the Fixed Effects Model. 3. Ease at which the results can be interpreted by a non-econometrician. For this reason, PSE included the True Impact Test and the OLS Model in this report. All three tests show similar results. 3.0 PSE included all of the same variables in the Fixed Effects Model that were included in the OLS model. Given the nature of the Fixed Effects Model, however, the intercept term and the treatment indicator term do not vary over time by household and were thus unable to be estimated. The equation used in the estimation is as follows: The parameters have the same interpretations as they did in the OLS Model. Of interest to this report is the parameter estimate of d. Again, this is the estimated impact on average daily use for those households in the treatment group currently receiving Home Energy Reports. The parameter estimate for d is equal to -0.637, revealing that those households who are being treated are, on average, reducing average daily usage by 0.637 kwh. As was the case for the OLS Model, the parameter estimate was highly significant with a t-statistic of 23.46. Estimated Impacts OLS Model Daily kwh -0.637 Percent Reduction -2.10% Connexus OPower Energy Measurement-Verification Report 11 MN0480806 8/9/10
4.0 Conclusion
4.0 4.0 Conclusion Given the stability of the results derived from three independent estimation procedures, the large sample size, and the randomness of sample selection, PSE concludes there are tangible energy savings resulting from this program. Results are robust with a high degree of confidence attached to them. The estimated savings range from 0.621 to 0.637 regarding kwh average daily use. In percentage terms, this is a range of 2.05 to 2.10 percent reductions in electricity use for those receiving the Home Energy Reports relative to those that are not. PSE also concludes there are higher savings resulting from receiving monthly reports relative to quarterly reports. Using a rolling start date OPower claims energy savings of 2.27 percent over the life of the pilot program. PSE used a start date to March 2009 for the fixed effect model and it produced a savings of 2.10 percent. PSE can verify a savings between 2.05 and 2.10 percent for the first year. Connexus OPower Energy Measurement-Verification Report 13 MN0480806 8/9/10
5.0 Appendix
5.0 5.0 Appendix The Appendix begins with a list of further research options that PSE believes would be worthwhile investments in uncovering the fullest potential of this program. The second part presents the models for the OLS Model and Fixed Effects Model discussed in this report. 5.1 Thoughts on Further Research Listed below are research items which could significantly enhance the understanding and optimal structuring of the program. 1. Diminishing returns and legacy impacts: Determining the incremental gains for each additional year of treatment would be of great interest. At what point do energy savings level off? Do savings actually decrease as customers become desensitized to the reports over time? Another important question is: Do savings continue after a household is discontinued from the program and how long do these legacy savings last? Would cycling households on and off the program be a cost effective strategy? 2. Interactions with energy efficiency programs: How does this program interact with other energy efficiency programs? Does it enhance or detract from them? Separating out the interaction impacts would very useful in determining accurate estimates of energy efficiency program savings. 3. Behavioral changes behind energy savings: An issue of this program is what is causing the decline in energy use for residences involved in the program. Is it a conservation effect, whereby households are more alert to turn off lights and adjust thermostats? Or are they more inclined to purchase energy efficient appliances and light bulbs? 4. Demographic relationships with energy savings: What types of households are more or less inclined to change energy use behavior? Is program targeting a valuable option? 5. Differences in energy savings due to neighbor comparisons Boomerang Effect : How does the comparison between the households use and the neighborhood energy use impact energy savings? Do those households who are classified as energy efficient relative to their neighbors actually increase energy use? What comparison percentages drive the largest energy reductions? Connexus OPower Energy Measurement-Verification Report 14 MN0480806 8/9/10
5.0 5.2 Model Results OLS Model Dependent Variable: kwhdur Independent Estimated Standard t- Variable Coefficient Error Statistic constant 20.92733 3.44746e-002 6.07036e+002 testind -6.55090e-002 2.79682e-002-2.34227 post 0.40916 3.58424e-002 11.41544 posttst -0.62150 5.01023e-002-12.40453 hdddur 0.23123 8.39772e-004 2.75350e+002 cdddur 2.60318 7.17852e-003 3.62635e+002 Number of Observations 2585514 R-squared 5.11016e-002 Corrected R-squared 5.10997e-002 Sum of Squared Residuals 8.99848e+008 Standard Error of the Regression 18.65570 Durbin-Watson Statistic 1.76353 Mean of Dependent Variable 29.64004 Fixed Effects Model Dependent Variable: kwhdur3 Independent Estimated Standard t- Variable Coefficient Error Statistic post3-2.73721e-002 1.94173e-002-1.40968 posttst3-0.63681 2.71456e-002-23.45898 hdddur3 0.22990 4.52831e-004 5.07703e+002 cdddur3 2.62359 3.87046e-003 6.77849e+002 Number of Observations 2585514 R-squared 0.16062 Corrected R-squared 0.16062 Sum of Squared Residuals 2.61489e+008 Standard Error of the Regression 10.05665 Durbin-Watson Statistic 1.85488 Mean of Dependent Variable -2.69004e-009 Connexus OPower Energy Measurement-Verification Report 15 MN0480806 8/9/10
5.0 5.3 PSE Background Founded in 1974, PSE is a full-service consulting firm serving the utility industry with offices in Wisconsin, Indiana, Minnesota, Ohio, and South Dakota. PSE has expertise in the areas of demand response, energy efficiency, revenue decoupling, merger valuations, load forecasting, cost and reliability performance benchmarking, T&D system planning and design, resource planning, communication technologies, smart grid investments, rate design, alternative regulation, and cost of service studies. 5.4 Authors Chris Ivanov, Economist Mr. Ivanov began his career at Wisconsin Public Power Inc. and is now a utility consultant. While at WPPI, he prepared, evaluated, and managed electric load forecasts, weather normalization, and small area forecasts for its 49 members. As a consultant, Mr. Ivanov prepares, evaluates, and manages electric load forecasts, surveys, and economic analyses for a wide range of clients including distribution and G&T cooperatives. His current focus is on assisting utilities with DSM studies and load forecasts. He has a Masters in Applied Economics from Marquette University and is currently finishing his MBA. Steve Fenrick, Economist Mr. Fenrick has nearly a decade of consulting experience in the utility industry. His work has focused on cost and reliability performance benchmarking, incentive regulation, demand-side management program designs and evaluation, revenue decoupling, and load forecasting. Mr. Fenrick has worked with cooperatives, investor-owned utilities, regulatory commissions, and international utilities. He earned a Bachelor of Science in Economics from the University of Wisconsin-Madison and is finishing a Masters degree in Agricultural and Applied Economics from the same university. Connexus OPower Energy Measurement-Verification Report 16 MN0480806 8/9/10