Niagara Mohawk Power Corporation d/b/a National Grid Residential Building Practices and Demonstration Program: Impact Evaluation Summary

Transcription

1 Niagara Mohawk Power Corporation d/b/a National Grid Residential Building Practices and Demonstration Program: Impact Evaluation Summary PROGRAM SUMMARY Prepared by: DNV KEMA, January 15, 2014 The OPower-administered Residential Building Practices and Demonstration (RBPD) Program utilizes a social marketing campaign, with normative messaging techniques, to encourage responsible energy behavior and choices. The campaign provides home energy reports (HERs) to households in National Grid-NY s combined gas and electric service territories in upstate New York. The HERs provide recipients with feedback on their household energy use including a comparison of the recipient household s energy usage with that of neighboring homes, thereby introducing a subtle form of peer pressure (often referred to as social norming ) among households to achieve energy savings. The NYS Public Service Commission (PSC) issued an Order on December 3, 2010, approving the RBPD program. EVALUATION OBJECTIVE AND KEY FINDINGS The evaluation compared ex ante and OPower savings estimates to those gathered through the billing analysis. The evaluation covered April 2011 through December 2012 and had the following key research questions: How well did the randomization match participating and control households? The randomization performed by OPower worked very well. The evaluation confirmed that the participating and control households had nearly identical pre-program energy use. Surveyed participating and control households had similar income, education, number of occupants, changes in number of occupants, home tenure, home ownership, home types, heating fuels, and water heating fuels. Electric-only households (including RBPD participating and control households) had high electricity use about 18,000 annual kwh. What energy savings resulted from the program? After removing point estimates of joint savings, DNV KEMA calculated energy savings for the period April 2011 through December 2012 of 44,844,098 kwh and 1,047,978 therms. This amounts to electricity savings of 2.43% per electric only household and 1.34% electric savings for dual fuel households. Gas savings were equivalent to 0.82% per household. When using the same number of accounts, OPower s estimates were similar: 43.8 MWh and 1,015,520 therms. For the period January 2012 through December 2012, DNV KEMA estimated savings of 28,618,739 kwh and 669,933 therms. What savings can be expected during a year with normal weather? The weather normalized savings for January to December 2012 were 26,871,561 kwh and 630,960 therms. This is equivalent to electricity savings of 2.30% per electric only household and 1.52% electric savings per dual fuel household. Weather normalized gas savings are 0.77% per household. Did the program increase participation in other incentive programs? Participation in other National Grid incentive programs increased. Based on incentive program tracking data provided by National Grid, RBPD participating households increased participation in other National Grid incentive programs at a statistically significantly greater rate than the control households(participant:3.3% versus control: 2.9%). Based on survey responses, there were no

2 statistically significant differences between RBPD participating and control households participation in NYSERDA upstream CFL rebate programs. The estimated savings from the participation increase were not statistically significant, but OPower evaluations typically remove point estimates of joint savings to stay as conservative as possible. We followed this convention. We estimated total joint savings from traditional incentive programs of about 152 MWh and about 53 MWh joint savings from the NYSERDA upstream CFL program (out of about 45,000 MWh) for the entire evaluated period of April 2011 through December Our estimates of gas joint savings from traditional incentive programs were actually negative, so we assumed they were zero. We therefore conclude that the current savings from the RBPD program are almost entirely from sources other than increased participation in other incentive programs. The reports did not heavily emphasize the limited number of available programs because those programs were already well-subscribed. What behaviors and non-incented measures contributed to savings? Based on survey results, electric savings primarily came from no cost/low cost behaviors such as turning off lights more often and turning down air conditioning at night. Relative summer and winter month gas savings indicate most (about 70%) gas savings come from water heating and cooking related behaviors or measures rather than space heating (about 30%). How did participating households react to the HERs? Almost all recall receiving the HERs (95%), about half (51%) spend 4 or more minutes reading, about one third (36%) thought the HERs were useful, and about eight percent spend 10 or more minutes reading. Table 1. Net Program Impact - Jan Dec 2012 Parameter Electric Energy (MWh/yr) Electric Demand (MW) Natural Gas (MMBtu/yr) Ex Ante Tracked Savings 1 14,580 n/a 116,478 Evaluation Realization Rate (RR) n/a 0.58 Evaluation Net-to-Gross Ratio (NTG) 1.0 n/a 1.0 Ex Post Net Impact 28,618 n/a 66, From Dec 3, 2010 NYS PSC Order authorizing the RBPD program. 2. Realization rate was calculated as ex post net impact divided by ex ante tracked savings. DETAILED FINDINGS: REALIZATION RATE AND NET-TO-GROSS Ex ante tracked savings were documented in the PSC order authorizing the RBPD program. Realization rate was calculated by dividing the ex post net impacts by the ex ante tracked savings. As can be seen in Table 1, the ex ante electric estimates were low and ex ante gas estimates were high. This evaluation did not investigate net-to-gross ratios it was assumed to be 1.0 for this program due to the use of a control group. EVALUATION METHODS AND SAMPLING According to National Grid records, the RBPD program started with 131,303 participating households and 65,646 control households in January of These households were selected in January, and HERs began going out in April. National Grid provided DNV KEMA with billing records for the still active households as of March 2013: electric billing records for 109,567 participant and 54,769 control households and gas billing records for 84,471 participant and 42,203 control households. The changes 2

3 from January 2011 to March 2013 were for participating and control households that no longer had active accounts with National Grid, most likely because they moved out of their residence. Participant households that opted out of receiving the HERs, but which still had active accounts with National Grid were included in the participant group for our analyses. 1 DNV KEMA s approach to estimating savings was to compare the pre-program and post-program energy use of participating households to those of the control households. OPower randomly assigned households to the participant and control groups, so subtracting out the control group s pre-post difference removes non-program effects. The basic estimate used a standard fixed effect, difference of difference regression model, without weather variables, that captured program savings for the specific time period. The average daily use (calculated by monthly use number of days in month) was regressed with fixed effects across customers and across all billing periods. DNV KEMA added heating degree days and cooling degree days variables to the basic model to account for the effects of unusually warm or cool weather. Savings partially attributable to other National Grid energy efficiency programs ( joint savings) were also computed. Point estimates of the joint savings were subtracted out when they were greater than zero. However, none of the joint savings estimates achieved statistical significance at the 90% confidence level. DNV KEMA completed a survey with 814 participant and 810 control households to determine which behaviors contributed to the observed energy savings and how participating households responded to the HERs. RECOMMENDATIONS AND PROGRAM ADMINISTRATOR RESPONSE The following recommendations were made by the evaluators conducting this study. Niagara Mohawk Power Corporation d/b/a National Grid s initial response to these recommendations is also summarized below and will be tracked over time. Recommendation 1: (HIGH) For claimed savings, use monthly, quarterly, or annual savings estimates provided by OPower based on comparisons to a control group and actual billing data (ex post estimates), and periodically verify and true up those savings using a third party evaluator. The billing analysis conducted by DNV KEMA revealed the following: Ex ante savings were not confirmed by the evaluation. The ex ante estimates were based on early OPower evaluations and were too low for electricity and too high for gas. When using the same number of accounts DNV KEMA s and OPower s ex post estimates were very similar. 1 This is often referred to as an intent to treat design. This design eliminates self-selection effects and increases the confidence that the resulting savings are due only to the decision to put a household into the participating group.

4 Regularly timed ex post estimates comparing a participant and control group are the most accurate way to calculate savings. In particular, regularly timed ex post estimates using control groups automatically factor in population characteristics (e.g. average energy use), weather (e.g. unusually warm/cool weather), and savings persistence which can reduce the accuracy of ex ante estimates. Thus we recommend that the program use OPower s ex post savings estimates as the program s official savings claims.. OPower reports savings to National Grid on a monthly basis using ex post estimates that do not rely on forecasting. In addition, to help detect and correct computation issues, we further recommend that a third party evaluator periodically review the savings estimates for accuracy. Official savings claims could be adjusted or trued up based on the third party results. This practice is likely to produce more accurate savings claims than ex ante estimates. PA Response to Recommendation 1: National Grid and Opower have previously come to the same conclusion that calculating and reporting ex ante program savings does not provide the best data to indicate true program performance. Thus, Opower is currently using actual billing data to calculate and report actual (ex post) program savings to National Grid. National Grid has recommended to DPS that these ex post savings be reported to DPS as part of National Grid s regular reporting process. Awaiting DPS s formal response. National Grid will have a third party evaluator review the actual (ex post) savings results when the next evaluation cycle is due. If the third party evaluator detects and substantiates any variance in claimed savings, National Grid will perform a trued up based on those third party results. Recommendation 2: (HIGH) Decide in advance how to handle finaled accounts, document this decision, and share the documentation with relevant parties. Customers are going to move out of their homes and close their accounts. National Grid should work with OPower to determine a method to identify this customer attrition. Additionally, overall savings estimates are somewhat sensitive to how savings for closed accounts are calculated. Therefore, National Grid and OPower should decide and document how to calculate savings with regard to closed accounts. Any future process evaluations should include an investigation of this procedure. Response to Recommendation 2: National Grid and Opower are currently exploring several options to determine the best method to identify Finaled accounts. Once the best method is determined, Finaled accounts will become part of the criteria for calculating program savings. When a solution is determined for Finaled accounts, National Grid will update our formal response to document the solution. 4

5 Future evaluations will include a review of the procedure for determining Finaled accounts and incorporating them into the savings calculation. Recommendation 3: (MEDIUM) Consider using future evaluations to investigate researchable questions beyond ex post savings, such as persistence of savings. While the national body of evidence from evaluations of first-year savings for OPower programs is consistent, that evidence is no longer completely relevant to future RBPD evaluations because the program is currently in its third year and will be in at least its fourth year before another evaluation occurs. Thus, third party validation of ex post savings estimates provided by OPower would still be prudent. However, evaluation of a more mature OPower program presents an opportunity to investigate additional unresolved questions such as savings persistence and longer term weather effects. National Grid should consider investigating some of these outstanding questions with future evaluation efforts, budgets permitting. PA Response to Recommendation 3: National Grid agrees that it would be beneficial to study the effects of a more mature program to learn more about persistence, weather effects, and the continuing savings potential for participants who have been in the program for a number of years. Budget permitting, these attributes would be included in the scope of work performed by a third party evaluation of the RBPD program. Recommendation 4: (MEDIUM) Future evaluations should consider using techniques to improve survey response rates, such as advance letters, rewards, and shortening the surveys. Advanced letters could increase the likelihood that respondents answer the phone when surveyors call. If used, advance letters should include National Grid logos and/or be printed on National Grid letterhead to maximize the likelihood of being read. They should state that the household will receive a phone call and include the originating name and number that will show on caller id. Rewards could increase the chance that a respondent who answers the phone agrees to complete the survey. In both cases, the evaluation team and National Grid should make sure they understand the legal requirements of the area where the program participants live. The evaluators should shorten surveys as much as possible because respondents are more likely to complete shorter surveys. PA Response to Recommendation 4: National Grid agrees that improving the survey response rate could help to gather more data to inform future evaluation results. National Grid would be interested in exploring industry best practices including advanced letters and rewards to improve response rates. Adopting specific practices would be driven by strategic analysis, legal requirements and budgetary constraints. National Grid will work with future third party evaluators to design a shorter survey in hopes of gaining more data through a greater number of completed surveys. Recommendation 5: (MEDIUM) To gather better data about behavior and equipment purchase changes caused by the HERs, National Grid and future evaluators should consider data collection strategies that occur during the course of the time period under evaluation. The accuracy of the current evaluation s survey responses depends on respondents memory of what they did two years ago. The surveys took place in 2013 and asked respondents what they did during Surveying a sample of

6 the participants and controls more frequently during the period being evaluated could yield more proximal and accurate data than two year retrospectives. PA Response to Recommendation 5: National Grid and Opower agree that surveying respondents in a more timely fashion would likely produce more current and accurate customer data which would be advantageous to the program. National Grid s latest contract with Opower now includes a feature called the Customer Engagement Tracker (CET) whereby Opower surveys 2,000 program participants twice per year to assess their sentiment towards the RBPD program and the utility in general. Opower has just completed the first CET and has prepared the results. Those results will be shared with National Grid at a meeting in November Recommendation 6: (LOW) Use the HERs to more prominently advertise the other energy efficiency programs available to participants. Program administrators at National Grid reported that the HERs did not aggressively promote the other energy efficiency programs because those programs were already well-subscribed. (Note: National Grid customer service representatives did direct callers to National Grid and NYSERDA programs if customers asked how to save energy.) Participation in the other programs might be slightly increased by more prominent advertising through the HERs. This practice would increase the likelihood of producing joint savings. Evaluations of other similar programs usually produce joint savings estimates of less than one percent. While these are relatively modest savings, the wide distribution of the HERs could still result in substantial overall savings. PA Response to Recommendation 6: Based on the limited number of residential energy efficiency programs offered in the period, and the relative success of those programs, National Grid did not find a need to actively promote those programs via the HERs. Programs were more passively promoted by linking the program to a related energy saving tip. However, with new program measures recently approved by the DPS, National Grid will coordinate between internal Program Managers responsible for those measures and actively promote programs that are in need of increased participation. We are also considering promotion of the NYSERDA home energy assessment program via the HERs. National Grid will schedule a meeting to discuss with NYSERDA in Q Recommendation 7: (LOW) National Grid and OPower could work together to implement a variation of the HERs that tests whether the comparisons to neighbors or comparisons to recipients own household are more important. OPower s program theory is that the comparisons to neighbors activate social norms in participating households which cause them to reduce their energy use. Our evaluation found some evidence to suggest that participants may be paying more attention to the comparisons to their own household. OPower could test the relative impact of each part of the HERS by 6

7 adding a second type of HER that omitted the social comparisons and retained the self-comparisons. The effects of each type of HER relative to the control group would help answer which part of the HERs is more important. PA Response to Recommendation 7: National Grid and Opower are currently conducting a similar study with the utility s RBPD program in Rhode Island. In this study, certain participants receive a HER that does not include a neighbor comparison. Results of this exercise will be available in Depending on those results, National Grid may consider conducting a similar study for upstate NY RBPD program.