Home Energy Reporting Program Evaluation Report. June 8, 2015

Transcription

1 Home Energy Reporting Program Evaluation Report (1/1/ /31/2014) Final Presented to Potomac Edison June 8, 2015 Prepared by: Kathleen Ward Dana Max Bill Provencher Brent Barkett Navigant Consulting Navigant Consulting, Inc.

2 Disclaimer: This report was prepared by Navigant Consulting, Inc. ( Navigant ) for Potomac Edison based upon information provided by Potomac Edison and from other sources. Use of this report by any other party for whatever purpose should not, and does not, absolve such party from using due diligence in verifying the report s contents. Neither Navigant nor any of its subsidiaries or affiliates assumes any liability or duty of care to such parties, and hereby disclaims any such liability Potomac Edison HER Program Evaluation Report Draft Page i

3 Table of Contents E. Executive Summary... 1 E.1. Program Description... 1 E.2. Key Findings... 1 E.2. Recommendations for Program Improvement Introduction Program Description Evaluation Objectives Evaluation Approach Statistical Consistency of the Program with an RCT Gross Impact Evaluation Methodology Uplift Analysis Methodology Net Impact Evaluation Data Used in Impact Analysis Gross Impact Evaluation Gross Impact Parameter Estimates Uplift of Savings in Other energy efficiency programs Verified Gross Program Impact Results Net Impact Evaluation Findings and Recommendations Potomac Edison HER Program Evaluation Report Draft Page ii

4 E. Executive Summary E.1. Program Description Potomac Edison s (PE) Home Energy Reporting (HER) program is designed to generate energy savings by providing residential customers with sets of information about their specific energy use and related energy conservation suggestions and tips. The information is provided in the form of Home Energy Reports that give customers various types of information, including: a) how their recent energy use compares to their energy use in the past; b) tips on how to reduce energy consumption, some of which are tailored to the customer s circumstances; and c) information on how their energy use compares to that of neighbors with similar homes. In other studies, this type of information has shown that customers are stimulated to reduce their energy use, creating average energy savings in the 1% to 2% range, depending on local energy use patterns. E.2. Key Findings The HER program savings for calendar year 2014 are presented in Table E-1. Findings include: Total verified gross program savings were 22,084 MWh. Savings are at the customer level and do not account for line losses. On average, participants reduced their electricity usage by 1.63%. The HER program increased participation in the Energy Efficient Products, Home Performance with Energy Star, Energy Efficient HVAC and DHW Equipment, and Appliance Turn-In programs. Total double counted savings are estimated at 221 MWh and are excluded from verified gross program savings. Double counted savings account for 1.0% of HER program electric savings. Savings generated by the HER program are within the typical range of savings for residential HER programs, which typically range from 1% to 2%. Table E Total Program Electric Savings Type of Statistic 2014 Number of Participants 75,600 Percent Savings, Electric 1.63% Verified Gross Electric Savings (MWh) Source: Navigant analysis. 22,084 E.2. Recommendations for Program Improvement Navigant found that savings are within the typical range of savings for residential behavior programs and therefore recommends that PE continue the program in its current form Potomac Edison HER Program Evaluation Report Draft Page 1

5 1. Introduction 1.1 Program Description PE s Home Energy Reporting (HER) program is designed to generate energy savings by providing residential customers with information about their specific energy use and related energy conservation suggestions and tips. The information is provided in the form of Home Energy Reports (HER) that illustrate: a) how customers recent energy use compares to their energy use in the past; b) tips on how the customers can reduce energy consumption, some of which are tailored to each customer s unique circumstances; and c) information on how the customers energy use compares to that of neighbors with similar homes. In other studies, this type of information has stimulated customers to reduce their energy use, creating average energy savings in the 1% to 2% range, depending on local energy use patterns. PE contracted with Opower to implement the program. An important feature of the program is that it is a randomized controlled trial (RCT). Eligible customers are randomly assigned to a treatment (participant) group or a control (non-participant) group, for the purpose of estimating changes in energy use due to the program. The HER program was launched in October 2012, with the first reports generated on October 1, The initial deployment of the program includes 75,600 residential customers, with an additional 26,250 residential customers designated as controls. 1.2 Evaluation Objectives The primary objective of the analysis in this report is to determine the extent to which participants in the HER program reduced their energy consumption due to the program Potomac Edison HER Program Evaluation Report Draft Page 2

6 2. Evaluation Approach The evaluation approach Navigant employed in this analysis is consistent with the methodology described in the SEE ACTION report, 1 relying on statistical analysis appropriate for RCTs. This evaluation has three primary components: checking the allocation of customers to the treatment and control groups for consistency with an RCT, regression analysis to quantify program savings, and quantification of double-counted savings from participation uplift in other energy efficiency programs. This section describes these components in more detail. 2.1 Statistical Consistency of the Program with an RCT Navigant compared the monthly energy usage of the treatment and control groups during the 12 month period prior to the start of the program (October 2011 through September 2012). If the allocation of the households across the treatment and control groups is truly random, the two groups should have the same distribution of energy usage for each of the 12 months before the start of the program. For this analysis, Navigant compared the mean usage for each of the 12 months before the start of the program. The results of the analysis indicate that the allocation of program households across the treatment and control groups is consistent with an RCT design. Figure 2-1 depicts the average energy usage for treatment and control households for the 12 months prior to the start of the HER program. The blue line indicates the average energy usage for the control group and the red dashed line indicates the average energy usage for the treatment group. The two lines in each graph are nearly identical, indicating no difference in average usage patterns for the treatment and control groups. Navigant conducted a statistical test on the difference in the mean energy usage in each of the twelve months. Navigant found the difference to be statistically significant at the 90% confidence level in March and April Average usage for the treatment and control groups differed by approximately 0.3 to 0.35 kwh per day during these months, or 0.6% to 0.7% of average energy usage. The two groups usage differed by a similar amount in September 2012, although the difference was not statistically significant. As an additional check, Navigant conducted a regression analysis in which average daily usage in the pre-program was a function of monthly binary variables and a binary treatment variable. The parameter on the treatment variable was not significant at the 90% confidence level, indicating no statistical difference in energy use between the treatment and control groups prior to the start of the program. In light of these results, and as detailed in the next section, Navigant used a statistical method appropriate for use with RCTs to quantify the energy savings for the program. 1 Todd, A., E. Stuart, S.Schiller, and C. Goldman. Evaluation, Measurement, and Verification (EM&V) of Residential Behavior-Based Energy Efficiency Programs: Issues and Recommendations. Lawrence Berkeley National Laboratory. May Available at: 2 Note that using a 90% confidence interval we would expect on average one out of every ten months to have a statistically significant difference in average consumption, due to random chance Potomac Edison HER Program Evaluation Report Draft Page 3

7 Figure 2-1. Average Daily Energy Use during the Pre-Program Year Source: Navigant analysis 2.2 Gross Impact Evaluation Methodology Navigant estimated program impacts using two approaches: linear fixed effects regression (LFER) analysis applied to monthly billing data, and a simple post-program regression (PPR) analysis with lagged controls. We ran both models as a robustness check. Although the two models are structurally very different, both generate unbiased estimates of program savings in an RCT. The LFER model combines both cross-sectional and time series data in a panel dataset. The regression essentially compares pre- and post-program billing data for participants and controls to identify the effect of the program. The customer-specific constant term ( fixed effect ) is a key feature of the LFER analysis and captures all customer-specific effects on energy usage that do not change over time, including those that are unobservable. The fixed effect represents an attempt to control for any small systematic differences between the treatment and control customers that might occur due to chance. Specifically, Navigant estimated the following regression model: Equation 2-1. LFER Model ADC kt = α 0k + α 1 Post t + α 2 Participant k Post t + ε kt Where, ADCkt Postt Participantk α 0k = The average daily usage in kwh for customer k during billing cycle t. This is the dependent variable in the model. = A binary variable indicating whether bill cycle t is in the post-program period (taking a value of 1) or in the pre-program period (taking a value of 0). = A binary variable indicating whether customer k is in the participant group (taking a value of 1) or in the control group (taking a value of 0). = The customer-specific fixed effect (constant term) for customer k. The fixed effect controls for all customer-specific effects on energy usage that do not change over time Potomac Edison HER Program Evaluation Report Draft Page 4

8 α 1, α 2 ε kt = Regression parameters corresponding to the independent variables. = The cluster-robust error term for customer k during billing cycle t. Clusterrobust errors account for heteroscedasticity and autocorrelation 3 at the customer level. Average daily savings are indicated by the parameter α 2. Program savings are the product of the average daily savings estimate, the number of days in the post-period 4, and the number of participants. As with the LFER model, the PPR model combines both cross-sectional and time series data in a panel dataset, but it uses the post-program data only, with lagged energy use for the same calendar month of the pre-program period replacing the customer-specific fixed effect as a control for any small systematic differences between the treatment and control customers. In particular, energy use in calendar month m of the post-program period is framed as a function of both the treatment variable and energy use in the same calendar month of the pre-program period. The underlying logic is that systematic differences between control and treatment customers will be reflected in differences in their past energy use, which is highly correlated with their current energy use. Formally, the model is: Equation 2-2. PPR Model ADC kt = β 0 + β 1 Participant k + β 2t ADClag kt Month t + β 3t Month t + ε kt Where ADC kt and Participant k are defined as in the LFER model, ADClag kt is customer k s energy use in the same calendar month of the pre-program year as the calendar month of month t, and Monthj is a binary variable taking a value of 1 if the observation is in Month j and 0 otherwise. In this model β 1 is the estimate of average daily energy savings due to the program. The use of interaction terms ADClag kt Month t allows the effect of lagged energy use on current energy use to vary by calendar month. Estimated parameters β 2t and β 3t are specific to each month of the post-program period. 3 Ordinary Least Squares (OLS) regression models assume the data are homoscedastic and not autocorrelated. If either of these assumptions is violated, the resulting standard errors of the parameter estimates are likely underestimated. A random variable is heteroscedastic when the variance is not constant. A random variable is autocorrelated when the error term in one period is correlated with the error terms in at least some previous periods. 4 Savings accrue for participants with active accounts Potomac Edison HER Program Evaluation Report Draft Page 5

9 2.3 Uplift Analysis Methodology The HERs include energy saving tips, some of which encourage participants to enroll in other energy efficiency programs offered by PE. If participation rates in other energy efficiency programs are the same for HER participants and controls, the savings estimates from the regression analysis are already net of savings from the other programs, as this indicates the HER program had no effect on participation in the other energy efficiency programs. However, if the HER program affects participation rates in other energy efficiency programs, then portfolio savings differ from the simple summation of savings in the HER and energy efficiency programs. For instance, if the HER program increases participation in other energy efficiency programs, the increase in savings may be allocated to either the HER program or the energy efficiency program, but cannot be allocated to both programs simultaneously. 5 On the other hand, if the HER program generates negative participation in other energy efficiency programs a negative spillover as might happen, for instance, if the HER program encourages behaviors or actions that reduce the value to customers of participating in other energy efficiency program then there is no double counting of savings. These negative savings should be included as HER program savings, because they represent a downward bias in the statistical estimate of HER program savings. In other words, the negative spillover inappropriately lowers the counterfactual energy use or baseline against which program savings are measured, causing the estimate of HER program savings to be too low. Gross verified savings are equal to the program savings less uplift savings. Navigant used a difference-in-difference (DID) statistic to estimate uplift in other energy efficiency programs, in which the change in the participation rate in another energy efficiency program between the current year of the program and the pre-program year for the control group was subtracted from the same change for the treatment group. For instance, if the rate of participation in an energy efficiency program during the current year of the program is 5% for the treatment group and 3% for the control group, and the rate of participation during the year before the start of the HER program is 2% for the treatment group and 1% for the control group, then the rate of uplift due to the HER program is 1%, which is reflected the calculation (5%-2%)-(3%-1%)=1%. The DID statistic generates an unbiased estimate of uplift when the baseline average rate of participation is the same for the treatment and control groups, or when they are different due only to differences between the two groups in time-invariant factors, such as the square footage of the residence. Navigant examined the uplift associated with four energy efficiency programs: Energy Efficient Products, Home Performance with Energy Star, Energy Efficient HVAC and DHW Equipment, and Appliance Turn-In. 2.4 Net Impact Evaluation Due to the RCT design of the HER program, free-ridership and participant spillover are incorporated in the results of the regression analysis. The RCT design does not account for non-participant spillover; however, non-participant spillover is expected to be small for this type of program. 5 It is not possible to use this method to quantify double counting of savings generated by programs for which tracking data is not available at the customer level, such as upstream lighting programs Potomac Edison HER Program Evaluation Report Draft Page 6

10 2.5 Data Used in Impact Analysis In preparation for the impact analysis, Navigant cleaned the data provided by the HER program implementer, Opower. The dataset included 75,600 participants and 26,250 controls. Navigant removed the following customers and data points from the analysis: Observations with less than 20 or more than 40 days in the billing cycle Observations outside of the twelve month pre-program period or program period of 1/1/2014 through 12/31/2014 Outliers, defined as observations with average daily usage at least ten times larger or ten times smaller than the median usage. 6 The dataset used for the LFER analysis contained 75,600 participants and 26,250 controls. PE reads customers energy meters on a bi-monthly basis. Therefore nearly half of the observations in the monthly billing data were estimated reads. There are two obvious ways to handle this situation in an econometric analysis. The first is to simply use the estimated reads disregarding the estimated/actual read designation and the second is to combine consecutive bills to create observations that are based on actual reads. Both approaches generate unbiased estimates of savings in our regression models. The second approach is usually to be preferred, because the first approach has the effect of increasing the variance of the model due to the measurement error in the dependent variable, causing the standard errors to be larger. The significance of this issue depends on the quality of the estimates of bills whether the estimates are very close to the actual energy use for the month which is revealed by the precision of the estimate of savings. However, Navigant discovered that most customers had estimated reads for more than two months during the spring and summer months of Combining estimated reads with the subsequent actual read resulted in bills spanning more than two months sometimes four months. This introduces a source of heteroskedasticity in the model error term, which can bias estimates of standard errors. Rather than combining consecutive estimated and actual bills and addressing this issue via statistical methods, Navigant chose the first approach for dealing with estimated reads. As discussed in the next section, the precision on the savings estimate is quite good. 6 The median usage is 46.1 kwh per day. Observations with usage greater than 461 kwh or less than 4.61 kwh per day were excluded from the analysis Potomac Edison HER Program Evaluation Report Draft Page 7

11 3. Gross Impact Evaluation The LFER and PPR models generate very similar results for program savings, with LFER estimates slightly lower than PPR estimates. We use LFER results for reporting total program savings. Overall verified gross program savings for calendar year 2014, excluding double-counted savings, were 22,084 MWh Gross Impact Parameter Estimates Parameter estimates for the estimated models are presented in Table 3-1 and Table 3-2 below. Key findings include: The LFER Post*Participant parameter estimate is statistically significant at the 90% confidence level, as is the PPR Participant parameter estimate. The PPR Participant parameter estimate is statistically significant at the 90% confidence level. Section 3.3 explains the calculation of program savings. Table 3-1. LFER Parameter Estimates, Equation 2-1 Variable Coefficient t-statistic Post Post * Participant Source: Navigant analysis. Note: T-statistics greater than in absolute value indicate results are statistically significant at the 90% confidence level. 7 Savings are at the customer level and do not account for line losses Potomac Edison HER Program Evaluation Report Draft Page 8

12 Table 3-2. PPR Parameter Estimates Variable Coefficient t-statistic Participant ADClag*January ADClag*February ADClag*March ADClag*April ADClag*May ADClag*June ADClag*July ADClag*August ADClag*September ADClag*October ADClag*November ADClag*December January February March April May June July August September October November December Source: Navigant analysis. Note: T-statistics greater than in absolute value indicate results are statistically significant at the 90% confidence level. 3.2 Uplift of Savings in Other Energy Efficiency programs LFER program savings include savings resulting from the uplift in participation in other energy efficiency programs caused by the HER program. To avoid double-counting of savings, program savings due to this uplift must be counted towards either the HER program or the other energy efficiency programs, but not both programs. The uplift of savings in other energy efficiency programs was a small proportion of the total savings: 221 MWh or 1.0 % Potomac Edison HER Program Evaluation Report Draft Page 9

13 Table 3-3 presents the details of the calculation of the double-counted savings due to uplift in other energy efficiency programs. The programs included in the uplift analysis were the Energy Efficient Products, Home Performance with Energy Star, Energy Efficient HVAC and DHW Equipment, and Appliance Turn-In. Table 3-3. Estimated Double-Counted Savings from Uplift in Other Energy Efficiency Programs Median program savings (annual kwh per participant) Appliance Turn In energy efficiency HVAC & DHW Equipment Program energy efficiency Products HPwES 1, # HER treatment households 75,600 75,600 75,600 75,600 Rate of participation, 2014 (%) 0.79% 1.61% 2.77% 3.20% Change in rate of participation from pre-program year (%) 0.22% 1.58% 0.19% 1.42% # HER control households 26,250 26,250 26,250 26,250 Rate of participation, 2014 (%) 0.75% 1.47% 2.65% 3.06% Change in rate of participation from pre-program year (%) 0.11% 1.45% 0.05% 1.30% DID statistic 0.11% 0.12% 0.14% 0.12% Change in program participation due to HER program Statistically significant at the 90% confidence level? Savings attributable to other programs (kwh) Yes No Yes No 91,553 35,051 25,722 68,961 Source: Navigant analysis. Note: Median program savings are equal to the median kwh impact for HER participants during the post-period. The estimate of double-counted savings is surely an overestimate because it presumes participation in the other energy efficiency programs occurs at the very start of the program year. Under the more reasonable assumption that participation occurs at a uniform rate throughout the year, the estimate of double-counted savings would be approximately 111 MWh, half the estimated value of 221 MWh. The upshot is that double counting of savings with other PE energy efficiency programs is not a significant issue for the HER program Potomac Edison HER Program Evaluation Report Draft Page 10

14 3.3 Verified Gross Program Impact Results Table 3-4 presents verified gross savings results of the HER program. The table also includes savings from the previous program year (2013) to serve as a comparison. Savings for the 2014 program year are within the typical range for behavior programs. On average participants reduced their usage by 1.63% (90% confidence interval: 1.48% to 1.78%). Total verified gross program savings during calendar year 2014 were 22,084 MWh. 8 Verified gross savings are calculated via the following equation: Equation 3-1. Calculation of Verified Gross Savings Verified Gross Savings = α 2 Number of Program Days 1000 Double Counted Savings Where α 2 is the parameter from Equation 2-1 that indicates average daily impacts from the LFER model. The number of program days is the sum of the number of days during the program year that a participant s account is active and they are receiving reports. 9 Table 3-4. Gross Program Savings and Uplift of Savings in Other Energy Efficiency Programs Type of Statistic Standard errors are provided in italics Number of Participants 75,600 75,600 Sample Size, Control 26,250 26,250 Percent Savings Average annual savings per customer (kwh) Verified Gross Savings, Prior to Uplift Adjustment (MWh) Savings Uplift in other energy efficiency programs (MWh) (1) Verified Gross Savings (MWh, therms) Source: Navigant analysis. 1.13% 1.63% 0.13% 0.15% ,324 22,306 1,809 2, ,054 22,084 (1) Negative double counted savings indicate that the participation rate in the energy efficiency program is higher for the control group than the treatment group. The baseline is artificially low, resulting in an underestimate of HER program savings. 8 Savings are at the customer level and do not account for line losses. 9 Customers that opt-out continue to generate savings after they opt-out of the program Potomac Edison HER Program Evaluation Report Draft Page 11

15 4. Net Impact Evaluation A key feature of the RCT design of the HER program is that the analysis inherently estimates net savings because there are no participants who otherwise might have received the individualized reports in the absence of the program. While some customers receiving reports may have taken energy conserving actions or purchased high efficiency equipment anyway, the random selection of program participants (as opposed to voluntary participation) implies that the control group of customers not receiving reports is expected to exhibit the same degree of energy conserving behavior and purchases. Thus, there is no free ridership, and no net-to-gross adjustment is necessary. Therefore, Navigant applied a net-to-gross ratio of Potomac Edison HER Program Evaluation Report Draft Page 12

16 5. Findings and Recommendations This section summarizes the key impact findings and recommendations. Finding 1. The treatment and control groups had similar usage prior to the start of the program. Therefore Navigant employed a statistical method appropriate for use with RCTs to quantify the energy savings for the program. Finding 2. The program generated 22,084 MWh of electric energy savings during calendar year On average, participants reduced their electricity usage by 1.63%. The savings are within the typical range of savings for residential Home Energy Report programs, which typically generate savings in the range of 1% - 2%. Recommendation. Continue the HER program in its current form. 10 Savings are at the customer level and do not account for line losses Potomac Edison HER Program Evaluation Report Draft Page 13