Seattle City Light Home Energy Report Program Impact Evaluation

Transcription

1 REPORT Seattle City Light Home Energy Report Program Impact Evaluation Submitted to Seattle City Light May 9, 2016 Principal authors: Mike Sullivan, Senior Vice President Jesse Smith, Managing Consultant Dulane Moran, Managing Consultant Ankit Jain, Consultant Adriana Ciccone, Senior Analyst

2 Contents 1 Executive Summary Program Summary Key Findings Conclusions Introduction and Program Description Program Description Key Research Objectives Organization of this Report Methods Data Cleaning Data Management Intention to Treat Equivalence Testing Extension Wave Equivalence Results Legacy/Expansion Wave Equivalence Results Regression Analysis Joint Savings Analysis Extension Wave Impact Findings Per-home kwh and Percent Impacts Aggregate Impacts Precision of Findings Legacy/Expansion Wave Impact Findings Per-home kwh and Percent Impacts Legacy Cohort Seattle City Light Home Energy Report Program Impact Evaluation i

3 5.1.2 Expansion Cohort CPW Legacy Cohort CPW Expansion Cohort Precision of Findings Aggregate Impacts Conclusions Energy Savings Comparison of Vendor Performance Seattle City Light Home Energy Report Program Impact Evaluation ii

4 List of Figures Figure 1-1: Percent Savings in Electric Consumption by Cohort 2014 and Figure 2-1: Assignment of Accounts...10 Figure 3-1: Difference in Average kwh Usage by Month (Extension wave, )...17 Figure 3-2: 2014 Average Daily kwh Pretreatment Estimates of Differences in Consumption by Month, via Pooled Regression (Extension Wave T-C)...18 Figure 3-3: Legacy/Expansion Wave Pretreatment Control/Treatment Comparison Average Daily Pretreatment Usage by Cohort...20 Figure 3-4: CPW Expansion Matched Control Group Selection...21 Figure 3-5: Pre and Post Matching Difference in Pretreatment Usage CPW Expansion...22 Figure 3-6: CPW Expansion Pretreatment Usage Comparison, Pre-matching...23 Figure 3-7: CPW Expansion Pretreatment Usage Comparison, Post-matching...23 Figure 3-8: Joint Savings Analysis Process for Rebate Programs...27 Figure 3-9: PSE Joint Savings per Home from Upstream Programs (Years 1 through 4)...28 Figure 3-10: Energy Efficiency Program Uplift for Expansion Cohort...29 Figure 4-1: 95% Confidence Intervals Associated with Extension Wave Impact Estimates...35 Figure 5-1: 95% Confidence Intervals Associated with Legacy Cohort Impact Estimates...43 Figure 5-2: 95% Confidence Intervals Associated with Expansion Cohort Impact Estimates...44 Figure 5-3: 95% Confidence Intervals Associated with CPW Legacy Cohort Impact Estimates 45 Figure 5-4: 95% Confidence Intervals Associated with CPW Expansion Cohort Impact Estimates...45 Figure 6-1: Per-household Impacts by Vendor in Months after Treatment...51 List of Tables Table 1-1: Average Number of Homes by Cohort 2014 and Table 1-2: Average Household & Aggregate 12 Month kwh Savings Total by Cohort... 3 Table 2-1: HER Waves, Key Characteristics, and Initial Group Assignments... 8 Table 2-2: Legacy and Expansion Assignment Groups... 9 Table 2-3: Vendor Wave Cohort Assignment...10 Table 3-1: Calculation of Treatment Percentage by Bill Month (Extension Wave)...16 Table 3-2: HER Demographics Summary Statistics (Extension Wave)...19 Table 3-3: Fixed Effects Regression Model Definition of Terms...24 Table 3-4: Extension Wave LDV Impact Calculation Output...26 Table 3-5: Joint Savings Adjustments by Cohort Upstream Programs...29 Table 3-6: CES Uplift Estimates by Year and Cohort...30 Table 3-7: CES Uplift Estimates by Year and Program...30 Table 4-1: Extension Wave Impact Estimates with ITT Adjustment...32 Table 4-2: Per-Home Average Extension Wave Impact Estimates...33 Table 4-3: Extension Wave Impact Estimates with Adjustment for Dual Participation...33 Table 4-4: Extension Wave Aggregate Impacts...34 Table 5-1: Legacy Cohort Impact Estimates by Month...36 Table 5-2: Legacy Cohort Impact Estimates with Adjustment for Dual Participation...37 Table 5-3: Expansion Cohort Impact Estimates by Month...38 Table 5-4: Expansion Cohort Impact Estimates with Adjustment for Dual Participation...39 Seattle City Light Home Energy Report Program Impact Evaluation iii

5 Table 5-5: CPW Legacy Cohort Impact Estimates by Month...39 Table 5-6: CPW Legacy Cohort Impact Estimates with Adjustment for Dual Participation...40 Table 5-7: CPW Expansion Cohort Impact Estimates by Month...41 Table 5-8: CPW Expansion Cohort Impact Estimates with Adjustment for Dual Participation...42 Table 5-9: Legacy/Expansion Waves Aggregate Impacts...47 Table 6-1: HER Program Impact Summary CY2014 and CY Table 6-2: Comparison of City Light and PSE 2014 HER Program Impacts...50 Table 6-3: Comparison of Initial 18 Month Performance by Cohort...52 Equations Equation 3-1: Fixed Effects Model Specification...24 Equation 3-2: Lagged Dependent Model Specification...25 Seattle City Light Home Energy Report Program Impact Evaluation iv

6 1 Executive Summary This report presents the results of Nexant s evaluation of the energy savings impacts of Seattle City Light (City Light) Home Energy Report (HER) program in 2014 and The program is designed to cause a reduction in billed energy consumption by providing customers with a comparison of their electricity consumption with that of other similar households. Recommendations for actions to take to save energy are also provided. Seattle City Light s Customer Energy Solutions (CES) launched the HER program in October 2009 by selecting a third party implementation vendor who delivered the program to customers. In 2011 the original program was expanded and the implementer continued to actively supply reports 1 to participating homes through These two waves are referred to as Legacy (2009) and Expansion (2011) in this report. In 2014, CES selected a second implementation contractor to test whether a more customizable platform would increase customer engagement and program uptake of other CES residential energy efficiency programs as compared to the program offered in the Legacy and Expansion waves. This second vendor delivered its version of the HER stimulus to customers starting in June 2014 and continued through the end of To estimate the energy savings from these programs, Nexant developed process for calendarizing City Light s bimonthly billing data and used industry standard regression modeling techniques to estimate load impacts on a monthly basis over the 27 month period of interest. Savings estimates are presented in three formats: 1) Energy savings per home (kwh). These are presented on a daily and annual basis. 2) Percent savings per home. Percent savings is calculated by dividing average energy savings observed in the treatment group by the average energy consumption in the control group. 3) Aggregate savings. Aggregate savings are expressed in MWh and calculated by multiplying the average impact per home by the number of active accounts receiving home energy reports. Impact estimates are presented separately for Fiscal Year 2014 (October 2013 to September 2014), Calendar Year 2014 (January 2014 to December 2014), Fiscal Year 2015 (October 2014 to September 2015), and Calendar Year 2015 (January 2015 to December 2015). 1 The first vendor did not send HERs to participants in January or February of 2014 while the implementation contract was being renewed. Seattle City Light Home Energy Report Program Impact Evaluation 1

7 1.1 Program Summary To ensure the internal validity of the analysis, Nexant first divided the HER program into five distinct cohorts for analysis based on treatment start date, implementation vendor and geographical location within the city. Within the Legacy and Expansion waves, geographic consideration was incorporated into the program delivery and evaluation by separating homes in neighborhoods targeted by the Community Power Works (CPW) initiative from homes outside of the original CPW target areas. Table 1-1 lists the five cohorts in the analysis along with the average number of homes in the treatment and control groups during calendar years 2014 and Table 1-1: Average Number of Homes by Cohort 2014 and 2015 HER Program Wave Treatment Start Date Treatment Group Size Control Group Size 2014 Average 2015 Average 2014 Average 2015 Average Legacy Oct-09 12,688 11,911 12,743 11,969 CPW Legacy Oct-09 2,418 2, Expansion May-11 17,706 16,437 7,747 7,223 CPW Expansion May-11 7,055 6,557 3,411 3,142 Extension Jun-14 40,837 47,302 22,293 20,086 Total Accounts 80,704 84,485 46,540 42, Key Findings Table 1-2 shows the average kwh savings per participant treated household, by analysis cohort, for calendar years 2014 and 2015 (CY2014 and CY2015, respectively). These estimates include an intent-to-treat adjustment to the Extension group to account for the fact that not all homes assigned to treatment received the treatment. It is also important to note that the Extension mailings began in June 2014 so the CY2014 estimates include only six months of treatment. The estimates for all five cohorts reflect downward adjustments to remove joint savings resulting from incremental participation in CES upstream and downstream programs by the treatment group. Nexant s approach to estimating joint savings and eliminating possible double-counting of savings is discussed in Section Not every home assigned to the treatment group of the Extension wave received HERs initially, which is discussed more fully in Section 3.2. Table 1-1 estimates the number of homes that were actively receiving reports from the Extension wave vendor in 2014 and The number of homes assigned to the Extension treatment group was approximately 25% higher than the number assigned to treatment for the Legacy and Extension wave vendor. Nexant s analytical approach to address this aspect of program delivery in the impact estimation is discussed in Section 3.2. Seattle City Light Home Energy Report Program Impact Evaluation 2

8 This table also shows Nexant s estimates of the total reduction in electricity consumption by the HER program in 2014 and These values also reflect a downward adjustment to eliminate double-counting of savings from incremental participation of HER treatment group homes in other CES offerings. The energy savings values are at the meter level so they do not reflect line losses that occur during transmission and distribution between the generator and end-use customer. Table 1-2: Average Household & Aggregate 12 Month kwh Savings Total by Cohort HER Program Wave Average kwh Savings/ Home (CY 2014) Average kwh Savings/ Home (CY 2015) Aggregate MWh Savings (CY2014) Aggregate MWh Savings (CY2015) Percent Savings (CY 2014) Percent Savings (CY 2015) Extension ,782 7, % 1.4% Legacy ,370 4, % 3.6% Expansion ,670 6, % 3.7% CPW Legacy , % 2.0% CPW Expansion ,171 2, % 3.8% Total/Weighted Avg ,470 21, % 2.4% The Legacy, Expansion and Extension wave cohorts all showed significant electric savings; and the savings for the Legacy and Expansion cohorts are significantly above average savings compared to other HER deployments in North America. While the 2014 per-home savings for the CPW Legacy group are even larger than those of the Legacy and Expansion cohorts, it s important to keep in mind that this estimate is based on a comparison with a very small control group (approximately 350 homes); and as a result, the margin of error for the CPW Legacy impact estimates is extremely wide. Consequently the impact estimate for almost every single month for this cohort is not statistically significant (see Section 5.2). Further evidence of the unreliability of the CPW Legacy estimates is the sharp decrease observed in Though not as severe, the CPW Expansion group also has a fairly wide margin of error associated with its impact estimates. This group also has a relatively small sample size compared to the other groups only 3,411 control group customers. While the impact estimates for these customers are statistically significant, it is possible that the impacts are much lower or higher than the point estimates presented in Figure 1-1 (see Section 5.2). Results for the Legacy and Expansion groups remained roughly flat in 2015 compared to 2014, while results for CPW Legacy were more than halved and the CPW Expansion group dropped by roughly 30%. In addition to benefitting from a full 12 months of activity, the Extension cohort showed improved performance from 2014 to The Extension wave began in June 2014 so the CY2014 impact estimates only include six months of treatment Seattle City Light Home Energy Report Program Impact Evaluation 3

9 Figure 1-1 displays the energy savings for each of the five HER cohorts on a percentage savings basis. Similar to the absolute impacts shown in Table 1-2, the CPW Legacy group showed the largest percent savings in 2014, though as noted above the estimates for this group were not statistically significant. Figure 1-1: Percent Savings in Electric Consumption by Cohort 2014 and 2015 The average percent savings shown for the 2014 Extension cohort result in Figure 1-1 uses only the impacts from July through December - after treatment had started - while all other cohorts use the full calendar year. Percent savings is a more appropriate lens to view the Extension wave cohort as it controls for the number of months of exposure and creates a more levelized 2014 comparison with the Legacy and Expansion (2009 and 2011) groups. HER evaluations have consistently shown a gradual increase in the treatment effect as the duration of exposure increases. The <1% impact observed among the Extension cohort is fairly typical for the initial year of an HER deployment; the second year of treatment exposure demonstrates this gradual increase in HER effectiveness. The Legacy and Expansion cohorts had been receiving HERs for 3 to 5 years by 2014, so the larger treatment effect is expected. Section 6.2 compares and contrasts vendor performance in greater detail. Seattle City Light Home Energy Report Program Impact Evaluation 4

10 1.3 Conclusions HER exposure produced significant energy savings among treated homes during the period of investigation. Per-home kwh savings were above average compared to similar HER implementations in North America, despite modest per-home electric consumption in SCL service territory (~10,000 kwh per home annually) There is a relationship between duration of HER exposure and savings levels, with the average treatment effect growing gradually over time. The vendor responsible for the Legacy and Expansion cohorts achieved larger average impacts during the first 18 months of exposure than the Extension vendor. However, the winter weather conditions faced by the Extension cohort were far milder than the Legacy or Expansion cohorts experienced in their first 18 months. When weather differences for the first 18 months of the cohorts are controlled, the performance of the programs offered by the two implementers is not significantly different. In other words, different weather conditions as well as some underlying differences in the population treated make the vendor differences less significant than they first appear. The experimental design of the CPW Legacy and CPW Expansion cohorts was compromised. This combined with small control group sizes led to savings estimates that either were not statistically significant or were marginally significant. Point estimates from these groups should be interpreted with the wide margin of error in mind. Seattle City Light Home Energy Report Program Impact Evaluation 5

11 2 Introduction and Program Description This section presents a brief description of Seattle City Light s HER program as implemented and operated in City Light s territory. It is informed by program documentation, a review of previous evaluations, and an understanding of the nuances in program design developed through regular communication with City Light staff during the evaluation process. 2.1 Program Description City Light s HER program has evolved over the past six years to include three distinct treatment waves at different times which, were provided by different vendors using different methodologies for defining similar households and contained somewhat different messaging strategies. It is necessary to take account of these important differences in assessing the energy savings resulting from the overall program. In essence, the program involved a number of separate treatments and it is necessary to separately analyze the impacts of the different treatments in order to obtain an accurate aggregate estimate of the impact of the program. In 2009, Seattle City Light s Customer Energy Solutions (CES) 4 began providing Home Energy Reports (HER) to a subset of single family residential customers using a third party implementation contractor. This is referred to as the Legacy wave. In 2011, the program was expanded to a larger selection of customers. This is referred to as the Expansion wave. Finally, starting in June 2014, a second implementation contractor began delivering HERs to additional customers intended to test whether a more customizable platform offered by the selected contractor would increase customer engagement and program uptake of other CES residential energy efficiency programs as compared to the HER product provided by the first contractor. This effort referred to as the Extension wave program was also designed to more actively market an opt-in report delivery component and provide enhanced web and paper communication options. The three waves of HER enrollment had slightly different features, which include the following. Legacy 2009: An initial wave of approximately 40,000 single family residential customers was randomly assigned by the program vendor into treatment and control groups in nearly equal proportions. This wave excluded the bottom 25% of electricity users (by consumption) from treatment and control households. Treatment households received HERs developed by the first contractor six times each year corresponding with customers billing cycles. Expansion 2011: An additional wave of approximately 36,000 customers began treatment by the first contractor in Month of This wave included two subgroups: 1) Customers who 4 Formerly Conservation Resource Division (CRD) Seattle City Light Home Energy Report Program Impact Evaluation 6

12 were located in a geographical area targeted by SCL s Community Power Works (CPW) 5 weatherization program; and 2) customers located outside the CPW target area. For both the CPW and non-cpw expansions, the treatment and control group allocation was adjusted to increase the proportion of homes included in the treatment group, originally targeting a ratio of 2.5 to 1. After accounting for attrition in customers who were originally assigned to treatment and control, this resulted in a treatment to control ratio of 1.9 to 1 for the CPW expansion and 2.3 to 1 for the non-cpw expansion. The bottom 25% of the customers in the subgroup located outside the CPW target area was excluded from the treatment and control groups. However, no customers were excluded from the treatment and control groups located inside the CPW target area. The 2011 Expansion wave included several other idiosyncrasies that complicate the impact evaluation. During the expansion, homes that had been included in the initial (Legacy 2009) enrollment, but were located in the CPW geographic area, were reassigned to the CPW group. This includes a group of approximately 2,500 homes that were converted from the control group to the treatment group in 2009 to the CPW treatment group. CPW treatment homes also received specific, targeted messaging promoting home energy audits and weatherization upgrade assistance not provided to the Legacy (2009) wave and treatment customers located outside the CPW target area for the 2011 expansion. For the focus of this report CPW services were available throughout City Light s service territory and the CPW groups no longer received differential messaging, so treatment of CPW and non-cpw cohorts was very similar Extension: In June of 2014, an additional wave of approximately 78,000 customers began treatment by a second vendor s HER that provided City Light with more opportunities to customize messaging and promote uptake of other CES programs. The sample for this wave was selected and randomly assigned to treatment and control groups by an independent third party. The sample for this wave was stratified by electric consumption levels, an electric heat indicator, and low income status. A randomized reserve sample that matched the characteristics of the 2014 Extension cohort was also identified and used to backfill sample loss. This wave excluded the bottom 15% of electricity users from treatment and control groups. The backfill accounts were exhausted rapidly by the vendor because of clustering issues related to irregularities in meter read routes. Table 2-1 summarizes the key characteristics of the three enrollment waves and the average number of active accounts in each cohort at the beginning of the evaluation period. Customer counts will differ slightly from the initial count over the course of the evaluation due to natural customer attrition. In this evaluation, the attrition rate was approximately 6% to 8% per year across all treatment customers. 5 CPW was funded through a grant from the American Recovery and Reinvestment Act (ARRA) and focused initially on promoting audits and weatherization services to a specific, targeted geographic area in Seattle. Seattle City Light Home Energy Report Program Impact Evaluation 7

13 Table 2-1: HER Waves, Key Characteristics, and Initial Group Assignments Wave Target Treatment: Control Ratio Characteristics Treatment Group Size Control Group Size Legacy, to 1 Top 75% of electricity users. 15,155 13,090 Expansion, to 1 Top 75% of electricity users. Geographic and message differences for CPW subgroup until ,934 11,158 Extension, to 1 Top 85% of electricity users. Stratification on sub-variables 55, ,293 Total households 95,793 46,541 Because of the complexities related to the treatment of the 2011 Expansion wave program in the CPW area, the analysis was subdivided into five distinct cohorts and evaluated separately: Legacy, Expansion, CPW Legacy, CPW Expansion and Extension. Nexant used the same nomenclature as the previous HER evaluation; however, the composition of the groups was slightly different. 7 The modified group assignments were made possible by more detailed data provided by the contractor for this evaluation, which had not been available for previous efforts. Within each of these cohorts, a treatment group was matched with a control group for evaluation. The decision to evaluate these groups separately was driven by the difference in implementation date for the Expansion and Legacy waves, as well as the difference in pretreatment randomization between the CPW and non-cpw area. This latter point is of particular importance for this evaluation. As mentioned before, when the program was expanded in 2011: Accounts in the CPW geographic area were added to treatment and control groups; Some of the accounts in the CPW geographic area for the Legacy wave that had originally been assigned to control group were shifted to the CPW treatment group; and Accounts located in the CPW target area in the Legacy treatment group were reclassified into the CPW Expansion treatment. These adjustments compromised the treatment/control randomization for customers in the CPW area, which is explained in more detail in Section In total, there are nine distinct groups of customers in the program, detailed in Table 2-2. The customer counts shown in Table 2-2 reflect the number of assignments. By the analysis period the focus of this 6 Extension participant count reflects the homes assigned to treatment. Only a subset of homes were actively receiving reports in 2014 and In this evaluation the CPW Legacy cohort only included the 2,900 homes that begin receiving treatment in The 2,500 homes that were converted from control to treatment in 2011 were assigned to the CPW Expansion cohort. Seattle City Light Home Energy Report Program Impact Evaluation 8

14 report total counts in all cohorts had dropped due to the natural rate of account turnover in the City. Table 2-2: Legacy and Expansion Assignment Groups Group # Group Name Group Description Treatment Start Date Customer Count 1 Legacy Treatment Added October 2009 as treatment October ,087 2 Legacy Control Added October 2009 as control NA 17,043 3 Expansion Treatment Added May 2011 as treatment May ,821 4 Expansion Control Added May 2011 as control NA 9,816 5 CPW Legacy Treatment 6 7 CPW Legacy/Expansion Control CPW Expansion Treatment. Previously Control Added October 2009 as treatment Added October 2009 as control and remained in control Added October 2009 to control. Converted to Treatment in May 2011 October ,912 NA 418 May ,538 8 CPW Expansion Treatment Added May 2011 as treatment May ,470 9 CPW Expansion Control Added May 2011 as control NA 4, Extension Treatment Added July 2014 as treatment July , Extension Control Added July 2014 as control July ,661 The treatment and control groups were grouped to create the five aforementioned cohorts of treatment and corresponding control customers, as described in Table 2-3. The non-cpw Legacy and Expansion cohorts are straightforward enough, since they simply consist of customers who are outside of the CPW area that were added to the program in 2009 and 2011, respectively. The CPW Legacy cohort consists of CPW area customers who were enrolled in the program in 2009 and were not reassigned in The CPW Expansion cohort consists of CPW-area customers who were assigned to treatment in 2011 new treatment customers as well as CPW Legacy control group customers who were moved to treatment and CPW-area customers who were assigned to control 2011 new control customers as well as CPW Legacy control group customers. Group 6 (Legacy Control) is used twice in this assessment because they are capable of serving as control group customers for two different cohorts, CPW Legacy and CPW Expansion. Seattle City Light Home Energy Report Program Impact Evaluation 9

15 Table 2-3: Vendor Wave Cohort Assignment Cohort Treatment Group #s Control Group #s Legacy 1 2 Expansion 3 4 CPW Legacy 5 6 CPW Expansion 7 and 8 6 and 9 Extension Figure 2-1 shows the assignment of accounts to the Legacy, Expansion, and Extension waves over time as well as the CPW designation of each group of homes. Figure 2-1: Assignment of Accounts Key Research Objectives The primary objective of the impact evaluation is to estimate changes in energy consumption attributable to the Legacy, Expansion, and Extension HER program treatments. Savings attributable to the programs are quantified on a per-home basis in both absolute (kwh) and relative basis (% savings) for each month. The evaluation also aggregates impacts to the 8 The Extension cohort was comprised of a primary group and backfill group that was used to supplement the primary group as it experienced attrition over the course of treatment. They are shown combined in Figure 2-1. Seattle City Light Home Energy Report Program Impact Evaluation 10

16 program level (MWh) by multiplying the average energy savings per treatment home by the number of participating homes. The following research questions guided impact evaluation activities: 1) Was the process used to select customers into treatment and control groups unbiased? 2) If issues in the randomization are detected, what corrective actions can be taken in the analysis to minimize the effects? 3) What were the energy savings associated with each wave of the home energy report program in fiscal years 2014 and 2015 and calendar years 2014 and 2015, and how do they differ from each other? 4) Is there evidence of seasonal effects associated with the energy savings? 5) How does removing double counting of savings from other energy efficiency programs affect HER savings estimates? To what extent does promotion via HER lift participation in other CES programs? 2.3 Organization of this Report This introduction is followed by: a section presenting the methods used to conduct the analyses; a section presenting results of the Extension wave impact analysis; a section presenting the results of the Legacy/Expansion impact analysis; and a final chapter presenting conclusions. Seattle City Light Home Energy Report Program Impact Evaluation 11

17 3 Methods The key objective of the HER impact evaluations is to measure the change in electric energy consumption (kwh) resulting from exposure to the normative comparisons and conservation messaging presented in the home energy reports. The approach for estimating HER impacts is built into the program delivery strategy. Eligible accounts are randomly assigned to either a treatment (recipient) group or a control group. The control group accounts are not exposed to HER in order to provide a valid statistical comparison group for estimating savings attributable to the HERs. In this randomized control trial (RCT) design, the only explanation for the observed differences in energy consumption between the treatment and control group is exposure to HER. While the program is designed to provide an equivalent control group to support comparison and impact estimates, decisions in program implementation can affect the reliability of these estimates. The relatively new Extension wave cohort appears to have equivalent treatment and control groups. The Legacy/Expansion cohorts with multiple waves and nonrandom reassignment have subgroups with apparent equivalence issues. Because the program was rolled out in two separate waves, the customers could not be pooled together for a single analysis. Instead, the program was subdivided into several different cohorts for analysis. This section describes the steps taken to create a valid counterfactual for each cohort before impact estimates were calculated. The impact estimates are based on analysis of bimonthly billing and program participation data provided by Seattle City Light. The RCT delivery method of the program removes the need for a net-to-gross analysis as the billing analysis directly estimates the net impact of the program. The billing regression estimates the total change in energy consumption in treatment group homes, which can include savings from incremental uptake of other CES conservation efforts in the treatment group claimed by other programs. A joint savings analysis is used to calculate the kwh that should be subtracted from the HER impact estimates to eliminate potential doublecounting. This report includes a downward adjustment to 2014 and 2015 HER impacts for upstream programs and joint savings. 3.1 Data Cleaning The HER impact evaluation relied on a large volume of participation and billing data provided by Seattle City Light. Key data elements include the following: Participant List complete lists of all City Light accounts that have been part of the Extension wave since June 2014 original and backfill and all participants from the Legacy (November 2009) and Expansion (April 2011) waves. This information was taken from datasets that also included the treatment/control assignment for each customer. Seattle City Light Home Energy Report Program Impact Evaluation 12

18 Billing History a bimonthly consumption (kwh) history for each account in the treatment and control group. Records included all billing cycles since assignment as well as the 12 months of pre-assignment usage history required for eligibility. This file also included the meter read date and the number of days in each billing cycle. Demographic Data demographic information for Extension wave participants was obtained from an account report spreadsheet, which included information such as inclusion in an elderly/low-income rate, geographic location, home type (single vs. multi-family), homeowner type (renter vs. owner), heating and cooling types, square footage of house, and vintage of house. Because similar demographic data was not available for Legacy/Expansion customers, demographic data was omitted from this assessment. Participation Tracking Data for Other CES Programs historical CES program participation of HER control and treatment group accounts for the joint savings analysis. These datasets were cleaned and combined as described below to form the analysis datasets for the Legacy, Expansion, and Extension wave evaluations Data Management In preparation for the impact analysis, Nexant combined and cleaned the participation and billing data provided by the City Light staff. In total, there were 6.2 million observations representing 164,478 unique customer and premise combinations across treatment and control groups. Nexant removed the following data points from the analysis: 808 observations associated with commercial accounts 14,000 observations that were associated with meter changes and were not usagerelated. 250 observations where a meter reached its maximum recording value and rolled over and where Nexant could not establish the use during the reading period. 821 observations where the duration of the meter read was less than a day, 15,000 observations where the meter read duration was greater than 120 days a fourmonth reading. 29,665 observations that were true duplicates for a given customer within a given billing period duplicates were dropped. 300,000 observations that were condensed because of a combination of estimated and actual readings being taken these were combined during the calendarization process and used to estimate the true usage for the account for that billing period and month. Like most electric utilities, Seattle City Light does not bill its customers for usage within a standard calendar month interval. Instead, billing cycles are a function of meter read dates Seattle City Light Home Energy Report Program Impact Evaluation 13

19 and vary across accounts. In some cases, Seattle City Light uses estimated meter reads for a billing cycle. To ensure customers were billed only for the electricity used, these estimated reads are then adjusted using subsequent actual meter reads so that the combined billing periods reflect the actual four or more month s consumption. In the cases where the estimated meter read overstated consumption, the subsequent billing cycle based on an actual meter read would be adjusted to net out the overestimated bill amount from the prior period. In these cases, neither the estimated consumption, nor the subsequent actual meter read values reflected true consumption during that billing period; however, the combined two-period consumption is accurate. In these cases, Nexant collapsed the data to obtain average use for the combined bill cycle period prior to analysis. Nexant s analysis of HER impacts is based on calendarizing bimonthly billing data. This process involves estimating average daily usage for each customer by month by using bimonthly bills, read date, and bill period duration. For each billing period, the average daily usage is assumed to be constant based on the billing usage amount for that cycle. The daily average for each calendar month is then calculated using these assumed daily consumption levels. Because meter read patterns are distributed similarly across the month within both treatment and control groups, using the bill month categorization does not introduce bias to the impact estimates. In the non-extension cohorts, four groups were identified based on geography and the timing of their randomization to treatment or control; customers that were randomized to treatment in 2009 in the Legacy and Legacy CPW waves and those that were randomized to treatment in 2011 in the Expansion and CPW Expansion waves. 3.2 Intention to Treat Not all accounts assigned to treatment consistently received HERs over the treatment periods included in the analysis. For the Extension wave, this was particularly relevant because a large portion of the homes originally assigned to treatment were discarded and replaced with backfill accounts early in the program rollout. Several programmatic considerations can prevent a treatment group home from receiving HER in a given month. Common reasons for an account not receiving an HER include the following: Postal Hygiene mailing addresses are subjected to deliverability verification by the printer. If an account fails this check due to an invalid street name, PO Box, or other issue, the home did not receive the HER mailer. Implausible Bill if billed usage for the previous month was unreasonably high or low, an HER was not sent. No Bill Received if the HER vendors did not receive usage data for an account from City Light within the necessary time frame to print and mail, the home did not receive an HER for the period. Seattle City Light Home Energy Report Program Impact Evaluation 14

20 Disruption in Meter Read Schedule changes to meter read routes in mid-2014 caused many customers to have some billing periods that were significantly longer or shorter than the standard of 60 days. When this occurred, the second vendor was sometimes unable to create reliable comparison reports and reports were not mailed.. These program delivery filters were not incorporated into the impact evaluation analysis dataset. This is necessary to preserve the RCT design because the filters are not applied to the control group in the same manner as the treatment group. Instead, Nexant employed an intention-totreat (ITT) analysis. In the ITT framework, the average energy savings per home assigned to the treatment is calculated via billing analysis. This impact estimate is then divided by the proportion of the treatment group homes analyzed that were active HER participants for a given month i.e., has received HER and is still an active customer. The underlying assumption of this approach is that all of the observed energy savings are being generated by the accounts receiving HERs. Nexant relied on the Extension contractor s monthly mailing counts for the numerator of the proportion treated calculation. The denominator of the proportion treated is the number of treatment group homes with billed kwh usage for the bill month. This calculation is presented by month in Table 3-1 for the study period for Extension wave customers. Because of the bimonthly billing schedule, customers only receive HER every other month. Therefore, in order to calculate the number of customers who received HER in a given billing cycle, the number of customers who received treatment was summed for two month periods. As shown in Table 3-1, Nexant used a one month lag to account for the time between mailing of reports and savings showing up in usage records. The average proportion treated was 75.5% in 2014 and 94.4% in Seattle City Light Home Energy Report Program Impact Evaluation 15

21 Table 3-1: Calculation of Treatment Percentage by Bill Month (Extension Wave) Report Month Number of Reports Mailed Treatment Group Size Number of Active Accounts % Treated Jun-14 21,542 1 Month Lag Applied Jul-14 22,098 55,514 43,640 Aug-14 16,936 54,891 Sep-14 16,240 54,239 33,176 Oct-14 22,194 53,713 Nov-14 23,502 53,246 45,696 Dec-14 21,774 52,874 Jan-15 21,540 52,545 43,314 Feb-15 24,878 52,224 Mar-15 24,645 51,867 49,523 Apr-15 21,299 51,403 May-15 28,771 50,937 50,070 Jun-15 28,392 50,413 Jul-15 20,557 49,617 48,949 Aug-15 20,702 49,233 Sep-15 27,144 48,842 47,846 Oct-15 29,727 48,480 Nov-15 14,382 48,154 44,109 Dec-15 24,105 47, % 61.5% 86.1% 82.7% 95.9% 98.8% 99.0% 98.3% 91.9% A similar approach was explored for the Legacy/Expansion cohorts, but the magnitude of the adjustment was minimal because the percentage treated was essentially 100%. Nexant reviewed the first vendor s initial mailing date and cumulative number of reports mailed values and calculated that between 99% and 100% of the active accounts in each Legacy/Expansion cohort were consistently receiving reports on a bimonthly schedule. 9 Nexant also used monthly participation counts, like the ones shown in Table 3-1, to estimate the aggregate impacts of the HER program by cohort. Per-home kwh savings estimates for each bill month are multiplied by the number of homes actively receiving HERs to arrive at the aggregate MWh impact achieved by the program. 3.3 Equivalence Testing Random assignment to treatment and control is expected to produce a situation in which the treatment and control groups are statistically identical on all dimensions prior to the onset of treatment; thus, the only difference between the treatment and control groups is exposure to 9 Except for Jan-Feb 2014 when no Legacy/Expansion wave reports were mailed Seattle City Light Home Energy Report Program Impact Evaluation 16

22 HER. The energy savings impact is therefore simply the difference in average electricity consumption between the two groups. The first step to assessing the impact of an experiment involving an RCT is to determine whether or not the randomization worked as expected. Several different methods exist to test the appropriateness of randomized treatment assignment. A simple test is to compare the frequency of treatment and control group customers across various demographic indicators. If randomization is done correctly, the proportion of customers assigned to treatment or control within a particular demographic category will be equal. A second method to confirm randomization is to look at the pretreatment data and compare the average energy use between treatment and control groups. A successful randomization will show no difference in pretreatment consumption between two effectively randomized populations, since the only difference between them will be a treatment that has not yet started. This section presents the results of the equivalence checks for the Extension and Legacy/ Expansion programs Extension Wave Equivalence Results Figure 3-1 shows the result of a difference in means calculation performed for HER accounts who had either been assigned to the treatment or control groups prior to receiving treatment. There were no obvious differences in energy consumption between treatment and control group by season, as evidenced by the fact the treatment and control groups usage patterns are almost indistinguishable. A plot of the difference between treatment and control, which is overlaid on a second axis in Figure 3-1, shows that the pretreatment difference is close to zero and does not systematically tend in the positive or negative direction. Figure 3-1: Difference in Average kwh Usage by Month (Extension wave, ) Seattle City Light Home Energy Report Program Impact Evaluation 17

23 Average Daily kwh The results from a linear random effects model run on pretreatment usage further indicate an appropriate randomization strategy. This data, shown in Figure 3-2, indicates the lack of any significant difference in consumption during the months before the treatment group began receiving home energy reports. Figure 3-2: 2014 Average Daily kwh Pretreatment Estimates of Differences in Consumption by Month, via Pooled Regression (Extension Wave T-C) Month of Pretreatment Period Table 3-2 provides additional summary information on the breakdown between treatment and control for different demographic indicators. As described above, a successful randomization should result in roughly equal proportions of customers in treatment and control within a demographic grouping. Table 3-2 shows the distribution of the prevalence of different groups within treatment and control assignments; in all cases randomization appears to have been successful. Based on these key demographics, there is no distinguishable difference between treatment and control groups. Any differences between these groups in the post-treatment period can be attributed to treatment and not any underlying difference between the groups. Seattle City Light Home Energy Report Program Impact Evaluation 18

24 Table 3-2: HER Demographics Summary Statistics (Extension Wave) Demographic Variable Control Treatment (Unit) Elderly (%) Low Income (%) Home Type Single Family (%) Multi-Family (%) Unknown (%) Heating Type Furnace (%) Heat Pump Hot Water (%) Electric (%) Stove (%) Other (%) Cooling Type Central (%) Window (%) Other/Unknown (%) Construction House Square Feet 2,245 2,281 (sq ft) Year Built (year) Legacy/Expansion Wave Equivalence Results As mentioned previously, due to complexities associated with the expansion of the Legacy/ Expansion program, particularly for the CPW-targeted area, this program was divided into four distinct cohorts that were evaluated separately. Figure 3-3 shows the pretreatment average daily use results for customers assigned to each of the four cohorts. With the exception of the CPW expansion cohort, the analysis found relatively small differences between treatment and control groups in the pretreatment period. The CPW Legacy cohort displays some sustained difference, but this is relatively small compared to the magnitude of the overall load note that the difference is plotted on a secondary axis in these graphs and can most likely be explained by the small control group for this cohort. While the Legacy, Expansion, and CPW Legacy customers appear to have relatively little pretreatment difference between treatment and control customers, the CPW Expansion group exhibits selection bias indicating nonrandom assignment. Without addressing this issue, the Seattle City Light Home Energy Report Program Impact Evaluation 19

25 Avg Daily (kwh) Difference (kwh) Avg Daily (kwh) Difference (kwh) preexisting difference between treatment and control customers could mistakenly be attributed to the impact of the HERs and could compromise the analysis. Figure 3-3: Legacy/Expansion Wave Pretreatment Control/Treatment Comparison Average Daily Pretreatment Usage by Cohort CPW Legacy CPW Expansion 2008m4 2008m m4 2009m10 Date 2009m m4 2010m m4 Date Control Treatment Control Treatment Difference T-C Difference T-C Legacy Expansion 2008m4 2008m m4 2009m10 Date 2009m m4 2010m m4 Date Control Treatment Control Treatment Difference T-C Difference T-C To address this issue, Nexant employed a matched-control group strategy for CPW Expansion customers only. This procedure sought to match customers in the treatment group with those in the control group who had similar average daily calendarized billing data for each month during the two years of the pretreatment period. By matching treatment customers to their counterparts in the control group that had similar annual load profiles, Nexant can then compare customers with similar usage behavior. Customers without a match between treatment and control were discarded, and the evaluation proceeded using only the customers with matched pretreatment usage, removing the preexisting difference between treatment and control. Any post-treatment difference in average daily usage after creating this matched-control group is attributed to the impact of the HER rather than pretreatment differences between the treatment and control groups. Figure 3-4 shows the improvement in the similarity of pretreatment usage for CPW Expansion customers across treatment and control assignments using the matched-control households. Seattle City Light Home Energy Report Program Impact Evaluation 20

26 Figure 3-4: CPW Expansion Matched Control Group Selection The difference lines in the above graphs are overlaid upon one another in Figure 3-5 to emphasize the change in pretreatment usage difference after the control group was adjusted using the matching procedure. After matching, the pretreatment difference is reduced but not eliminated. As described in the next section, the regression model chosen for these customers also minimizes the effect on the impact of treatment of observable differences between the treatment and control groups by controlling for any remaining post-matching differences in usage between the groups. Seattle City Light Home Energy Report Program Impact Evaluation 21

27 Figure 3-5: Pre and Post Matching Difference in Pretreatment Usage CPW Expansion Figure 3-5 presents estimated average daily kwh consumption by month for treatment and control customers in the CPW Expansion group. To ensure the overall distributions of this variable are similarly distributed across treatment and control, box plots showing the minimum, maximum, interquartile range, mean, and median of monthly usage data across treatment and control were created and shown in Figure 3-6 and Figure 3-7 for the pre and post-matched control groups, respectively. The increased similarity of the treatment and control distributions after matching gives further evidence that the randomization for the CPW datasets was improved by the matching process. Seattle City Light Home Energy Report Program Impact Evaluation 22

28 Figure 3-6: CPW Expansion Pretreatment Usage Comparison, Pre-matching CPW Expansion Pre-Matching C T C T C T C T C T C T C T C T C T C T C T C T Average Daily kwh excludes outside values Figure 3-7: CPW Expansion Pretreatment Usage Comparison, Post-matching CPW Expansion Post-Matching C T C T C T C T C T C T C T C T C T C T C T C T Average Daily kwh excludes outside values Seattle City Light Home Energy Report Program Impact Evaluation 23

29 3.4 Regression Analysis The Legacy, Expansion, and Extension wave evaluations used RCTs with large samples. While a difference in means would produce a reasonable estimate, regression modeling is better equipped to net out subtle underlying differences between the cohort treatment and control groups and produces a more precise impact estimate. Two different methods are typically used to achieve this objective: a linear fixed effects regression (LFER) model and a lagged dependent variable model (LDV). Each model is appropriate in different situations, which are discussed in this section. The basic form of the LFER model is shown in Equation 3-1. Average daily electric consumption for treatment and control group customers is predicted in a linear regression model including an indicator variable for the billing period of the study, a treatment indicator variable, and a customer-level indicator variable: Equation 3-1: Fixed Effects Model Specification kwh ity = customer i β i + t=1 y=2011 I ty β ty + t=1 y=2011 I ty τ ty treatment ity + ε ity Table 3-3 provides additional information about the terms and coefficients in Equation Table 3-3: Fixed Effects Regression Model Definition of Terms Variable Definition kwh ity Customer i s average daily energy usage in billing month t of year y. customer i β i I ty β ty treatment ity τ ty ε ity An indicator variable that equals one for customer i and zero otherwise. This variable models each customer s average energy use separately. The coefficient on the customer indicator variable. Equal to the mean daily energy use for each customer. An indicator variable equal to one for each monthly billing period t, year y and zero otherwise. This variable captures the effect of each billing period s deviation from the customer s average energy use over the entire time series under investigation. The coefficient on the billing period t, year y indicator variable. The treatment variable. Equal to one when the treatment is in effect for the treatment group. Zero otherwise. Always zero for the control group. The estimated treatment effect in kwh per day per customer in billing month t of year y; the main parameter of interest. The error term. The effect of the program for each month of the program (in kwh) is measured by the parameter τ ty. The main advantage to an LFER model is that it removes any preexisting differences between the treatment and control groups, so that it produces an accurate estimate of the treatment effect. This is the preferred method when there is evidence of bias in the pretreatment usage, Seattle City Light Home Energy Report Program Impact Evaluation 24

30 as was the case with several of the groups within the Legacy/Expansion waves. For the sake of consistency, all of the Legacy/Expansion cohorts were evaluated using the LFER model and each cohort was evaluated separately. The CPW expansion cohort required the use of a matched control group in conjunction with the LFER model. It is also possible, and sometimes more appropriate to estimate the treatment effect of the program using lagged dependent variable (LDV) model. This model is slightly different from the LFER model in that it incorporates individual heterogeneity by explicitly including past values of an individual s energy consumption as control variables on the right-hand side of the regression equation. In this case, we would modify Equation 3-1 as follows: Equation 3-2: Lagged Dependent Model Specification kwh ity = β + I ty β ty + I ty τ ty treatment ity + θkwh it 1 + ε it t=1 y= t= y=2011 In this specification, the intercept remains the same for everyone no i subscript on the β and there is an additional term kwh it-1 that represents the energy consumption for individual i in a previous period in this case, the average usage during that month in the year prior to the onset of the treatment. This is akin to saying that what makes consumers unique is captured entirely by their past levels of consumption. Equation 3-2 can be estimated by pooled ordinary least squares (OLS) regression, provided that there is no serial correlation in the error term and that there are no omitted variables that are correlated with the treatment. The underlying identification assumption is that average consumption without the treatment would be the same for both treatment and control customers. Unlike the LFER model, an LDV model can explicitly control for any measurable customer characteristics that vary over time. Another advantage is that pooled OLS uses both variation over time within customers and variation between customers so that the resulting estimates of the treatment effect will be relatively more precise than the LFER estimate. The main downside to the LDV model is that if fixed effects do indeed exist and are correlated with the treatment variable, then the LDV estimate of the treatment effect will potentially be biased. Additionally, the LDV model relies on being able to measure the relevant time-varying variables and cannot control for unobserved variables like the LFER model can. An LDV model is appropriate to use when the treatment and control groups show no systemic bias during the pretreatment period. In the aggregate, the two groups should show very similar pretreatment electricity usage, indicating internal validity in randomization. This was the case with the Extension wave program, as evidenced by the equivalence testing presented in Section Nexant estimated the LDV model for the Extension wave customers with the original and backfill customers pooled together, since the equivalence checks indicated that this was a valid treatment/control cohort. The model specification includes an interaction term between the treatment indicator variable and the indicator variable for the bill month term. This specification generates a separate estimate of the HER daily impact for each bill month. Table 3-4 illustrates Seattle City Light Home Energy Report Program Impact Evaluation 25

31 the calculation of monthly impact estimates from the regression model coefficients for homes assigned to treatment. Each month s coefficient is the treatment effect on average daily electricity usage. Table 3-4: Extension Wave LDV Impact Calculation Output Bill Month Daily Treatment Coefficient (τ) Average Daily kwh Impact Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec These impact estimates reflect the average daily treatment effect (in kwh) for the entire treatment group, even though some customers may not have started receiving treatment in a given month. These estimates of the average HER impact per assigned home were then divided by the proportion of customers treated, as shown in Table 3-1, to estimate the average treatment effect per participating home. 3.5 Joint Savings Analysis The regression model outputs and subsequent intention-to-treat adjustments discussed previously produce estimates of the total change in electricity consumption in homes exposed to HER. However, some portion of the savings estimated by the regression may be attributable to an increased propensity of HER treatment group homes to participate in other CES energy efficiency offerings than control group homes. The primary purpose of the joint savings analysis is to quantify annual kwh savings attributable to this incremental participation and subtract it Seattle City Light Home Energy Report Program Impact Evaluation 26

32 from the HER impact estimates. This downward adjustment prevents savings from being double-counted by both the HER program and the associated rebate program. A secondary objective of the dual participation analysis is to better understand the increased CES participation, or uplift triggered by inclusion of marketing messages within HER. The ability to serve as a marketing tool for other CES initiatives is an important part of what makes HER attractive to the City. Although the energy savings from the incremental energy efficiency participation are netted out of the HER savings to prevent double-counting, this increased uptake is desirable for the City and should be leveraged and encouraged as much as possible. Joint savings analysis falls into two broad categories based on the delivery mechanism of the program. For rebate programs, participation is tracked and can be linked to a City Light account. Using tracking records from these other programs, it is possible to calculate exactly how much incremental participation occurred in the treatment group and the claimed kwh savings attributable to the efficient installations. Figure 3-8 provides an overview of the required steps to calculate a downward adjustment to HER program impacts to account for joint savings. Figure 3-8: Joint Savings Analysis Process for Rebate Programs Match the data to the treatment and control homes by Customer Code and Premise ID Assign each rebate transaction to a bill month based on the participation date field in the tracking data Exclude any installations that occurred prior to the home being assigned to the treatment or control group Calculate the daily kwh savings of each rebated measure Sum the daily kwh impact by Account for measures installed prior to each bill month Calculate the average kwh savings per day for the treatment and control groups by bill month Calculate the incremental daily kwh from energy efficiency (treatment control) and multiply by the average number of days per bill month ( ) Joint savings analysis is more challenging for upstream programs because participation is not tracked at the customer level and therefore cannot be tied back to HER treatment and control group homes for comparison. While incremental uptake of upstream lighting measures by the Seattle City Light Home Energy Report Program Impact Evaluation 27

33 Joint Savings per Home (kwh/year) treatment group is certainly a reality of HER deployment, precisely estimating the modest difference in uptake is extremely resource intensive. City Light and Nexant elected to rely on the results of a robust joint savings analysis 10 conducted by neighboring utility Puget Sound Energy. PSE s upstream program offerings are similar to City Light s and the HER programs offered by the two utilities began at roughly the same time. Figure 3-9 shows the average incremental kwh savings attributable to upstream participation by the PSE HER treatment group from 2009 to Notice that the joint savings increase consistently over time with the exception of the year 6 findings. This is because savings from the efficient lighting installations are cumulative over the useful life of the product. As the HER treatment group continues to purchase efficient discounted products at a slightly higher rate than the control group the separation (in cumulative kwh) between the two groups grows. Figure 3-9: PSE Joint Savings per Home from Upstream Programs (Years 1 through 4) Upstream Program Joint Savings = * Number of Years Year of HER Deployment Nexant used a simple linear trend line to determine the slope of the upward trend and estimate an average kwh/year growth rate for upstream joint savings. Table 3-5 shows the joint savings adjustment Nexant applied to per-home impacts by duration of exposure. 10 PSE Home Energy Reports Impact Evaluation Seattle City Light Home Energy Report Program Impact Evaluation 28

34 Table 3-5: Joint Savings Adjustments by Cohort Upstream Programs Source HER Deploy ment Year Joint Savings (Annual kwh per Home) Fitted Value for Joint Savings Applied (Annual kwh per Home) 2014 Report Application 2015 Report Application Applied to Extension Cohort PSE 2014 Report Applied to Extension Cohort Applied to Expansion Cohorts Applied to Expansion Cohorts Applied to Legacy Cohorts Applied to Legacy Cohorts In addition to removing the impacts of upstream programs, Nexant also assessed the impact of SCL s customers participating in other energy efficiency programs. Program participation data for energy efficiency programs, as well as SCL-provided estimates of annual kwh savings per measure, were used to construct aggregate savings for each cohort to identify the extent to which receiving HERs lead to increased program enrollment. Figure 3-10 shows an example of the uplift associated with treatment for the Expansion cohort by month. The blue bars representing the difference in per-home daily kwh saved per day was calculated for each cohort and applied to the aggregate results to net out these joint savings. Figure 3-10: Energy Efficiency Program Uplift for Expansion Cohort Seattle City Light Home Energy Report Program Impact Evaluation 29

35 The increase in enrollment was also quantified by the uplift that treatment gave to savings in other energy efficiency programs. Table 3-6 shows the average daily kwh savings from CES programs for the treatment and control group of each cohort. The uplift is the percent difference between the claimed savings in the treatment group compared to the baseline savings from the control group. In both the Expansion and CPW Expansion cohorts, the treatment group showed a large uplift in program participation, while the other groups saw modest uplift. Table 3-6: CES Uplift Estimates by Year and Cohort Cohort Extension Legacy Expansion CPW Legacy CPW Expansion Average Customer Year Treatment Average Yearly kwh Saved Control Average Yearly kwh Saved Uplift % % % % % % % % % % % % Of the programs included in the joint savings analysis, the Solar program provided the highest savings for both treatment and control groups, roughly 140kWh to 215kWh per household per year. Table 3-7 gives the full set of program uplift and savings for treatment and control groups, split out by program. Table 3-7: CES Uplift Estimates by Year and Program Program Year Treatment Avg. Yearly kwh Saved Control Avg. Yearly kwh Saved Uplift Conservation Kit LED % % % % Refrigerator % Seattle City Light Home Energy Report Program Impact Evaluation 30

36 % Solar Upstream Lighting Washer/Dryer Water Heater All Programs % % N/A N/A % % % % % % Of the total incremental impact of treatment, the Solar and Upstream Lighting programs led to the highest savings for the average household. Shown below in Figure 3-11 is a visual representation of the annual kwh savings attributed to treatment across the various programs in 2014 and Savings due to programs other than the Solar or Upstream Lighting programs grew in relative contribution to total savings in 2015 as compared to Figure 3-11: Annual kwh Savings for the Average Household by Program Seattle City Light Home Energy Report Program Impact Evaluation 31

37 4 Extension Wave Impact Findings 4.1 Per-home kwh and Percent Impacts Nexant estimates the average participating Extension wave HER home saved 43.6 kwh of electricity from July 2014 to December This represents a 0.86% reduction in total electric consumption, compared to the control group over the same period. In 2015, with a full year of data, Extension wave customers saved kwh, or 1.44% reduction in consumption. These final estimates reflects an upward adjustment to account for the intention-to-treat methodology and a downward adjustment to prevent double-counting of savings attributable to incremental participation of treatment groups in City Light upstream energy efficiency programs. Table 4-1 shows the impact estimates in each bill month for the average home assigned to treatment, which were derived from an LDV model. The table also shows the subsequent adjustment to account for the fact that only a subset of homes assigned to treatment was actively participating in the Extension wave program during the study period. Table 4-2 shows the average savings per Extension customer summarized at the fiscal and calendar year level. Table 4-1: Extension Wave Impact Estimates with ITT Adjustment Month Average Customer Impact (Intent to Treat, kwh/day) Percent of Treatment Group Treated Average Customer Impact (Received Treatment, kwh/day) Control Group Usage (kwh/day) Percent Load Impact Jul % % Aug % % Sep % % Oct % % Nov % % Dec % % Jan % % Feb % % Mar % % Apr % % May % % Jun % % Jul % % Aug % % Sep % % Oct % % Nov % % Dec % % Seattle City Light Home Energy Report Program Impact Evaluation 32

38 Table 4-2: Per-Home Average Extension Wave Impact Estimates Month Average Customer Impact (Intent to Treat, kwh/day) Average Customer Impact (Received Treatment, kwh/day) Fiscal Year Calendar Year Fiscal Year Calendar Year An adjustment factor of 0.1 annual kwh per home for FY 2015 is applied to HER impact estimates in Table 4-3 to arrive at the final net verified program impact per home. Section 3.5 provides additional detail on the calculation of the 0.1 kwh downward adjustments for overlapping participation in other City Light EE programs. Table 4-3: Extension Wave Impact Estimates with Adjustment for Dual Participation Date Range Total kwh Savings in Treated Homes kwh Savings Attributable to Other Programs Net HER Impact Estimate kwh Annual Control Group Usage (kwh) Percent Savings Fiscal Year 2014 Calendar Year 2014 Fiscal Year 2015 Calendar Year , % , % , % , % 4.2 Aggregate Impacts The total impact of the Extension wave program in the City Light service territory is calculated by multiplying the per-home impacts (adjusted for ITT and joint savings) for each bill month by the number of participating homes. Over the six month period examined for calendar year 2014, Extension wave participants conserved nearly 1.8 GWh of electricity. Over the fiscal year 2014, 0.7 GWh of electricity was saved. These results increased in 2015, with 7.1GWh of electricity saved in CY2015 and over 6.0 GWh saved in FY2015. The aggregate impacts presented in Table 4-4 are at the meter level so they do not reflect line losses that occur during transmission and distribution between the generator and end-use customer. Seattle City Light Home Energy Report Program Impact Evaluation 33

39 Table 4-4: Extension Wave Aggregate Impacts Bill Month # of Customers Received Mailing Percent of Treatment Group Treated Per Home kwh Savings (Received Treatment) Aggregate MWh Impact for Treated Homes MWh Attributable to Other Programs Net Aggregate MWh Jul-14 43, % (0.43) 88 Aug-14 43, % Sep-14 33, % Oct-14 33, % Nov-14 45, % Dec-14 45, % Jan-15 43, % Feb-15 43, % (1.41) 479 Mar-15 49, % Apr-15 49, % May-15 50, % Jun-15 49, % Jul-15 49, % Aug-15 48, % Sep-15 48, % Oct-15 47, % Nov-15 44, % Dec-15 43, % , Fiscal Year 2014 Total 1, ,109 Calendar Year 2014 Total 1, ,779 Fiscal Year 2015 Total 6, ,044 Calendar Year 2015 Total 7, , Precision of Findings The margin of error of the per-home impact estimate ranged from 0.28 kwh per day to 0.56 kwh per day at the 95% confidence interval over the six month period in other words, there is a 95% chance the true value of the impact for a given month is within that range. Nexant clustered the variation of the LDV model by customer account to produce a robust estimate of the standard error associated with treatment coefficients. Figure 4-1 shows the confidence intervals around each estimate of HER impacts per household assigned to the treatment group. Impacts are small and generally not significant for the first five months of treatment; however, the impacts grow over time and remain significantly greater than zero. Seattle City Light Home Energy Report Program Impact Evaluation 34

40 Figure 4-1: 95% Confidence Intervals Associated with Extension Wave Impact Estimates Seattle City Light Home Energy Report Program Impact Evaluation 35

41 5 Legacy/Expansion Wave Impact Findings 5.1 Per-home kwh and Percent Impacts Per-home results were calculated for each cohort separately using the LFER model and are presented here. Joint savings attributable to City Light upstream energy efficiency offerings were then calculated and removed to avoid double-counting Legacy Cohort The average customer in the Legacy treatment cohort who received HERs in 2009 saved 423.3kWh during calendar year 2014, which equates to 3.56% of total consumption. Impacts in 2015 decreased slightly to 410.4kWh, but increased slightly to represent a 3.60% reduction in energy consumed. Estimated impacts remain fairly steady year over year, though winter impact peaks suggest seasonality in program impact. Table 5-1 shows the average daily savings per home on a monthly basis for the Legacy cohort. Table 5-1: Legacy Cohort Impact Estimates by Month Month Average Customer Impact (kwh/day) Control Group Usage (kwh/day) Percent Load Impact Oct % Nov % Dec % Jan % Feb % Mar % Apr % May % Jun % Jul % Aug % Sep % Oct % Nov % Dec % Jan % Feb % Seattle City Light Home Energy Report Program Impact Evaluation 36

42 Month Average Customer Impact (kwh/day) Control Group Usage (kwh/day) Percent Load Impact Mar % Apr % May % Jun % Jul % Aug % Sep % Oct % Nov % Dec % Fiscal Year Calendar Year Fiscal Year Calendar Year Table 5-2 shows the application of a joint savings adjustment for upstream EE and the percent savings in electric consumption per home for the Legacy cohort. Table 5-2: Legacy Cohort Impact Estimates with Adjustment for Dual Participation Date Range Total kwh Savings in Treated Homes Savings Attributable to Other Programs Net HER Impact Estimate kwh Control Group Usage (kwh) Percent Savings Fiscal Year , % Calendar Year , % Fiscal Year , % Calendar Year , % Expansion Cohort The average customer in the Expansion cohort who received treatment saved 376.7kWh during calendar year 2014, which equates to 3.34% of total consumption estimates were larger than in 2014, with a per-day kwh savings of 1.1 kwh (or 404.5kWh per household per year net of joint savings), a 3.73% reduction in electricity consumption when compared with the control group. Similar to the Legacy group s results, both absolute and percentage impacts peak during the winter month. However, overall, these estimates are in line with the magnitude of Seattle City Light Home Energy Report Program Impact Evaluation 37

43 the estimates for the Legacy cohort. Table 5-3 shows the average daily savings per home on a monthly basis for the Expansion cohort. Table 5-3: Expansion Cohort Impact Estimates by Month Month Average Customer Impact (kwh/day) Control Group Usage (kwh/day) Percent Load Impact Oct % Nov % Dec % Jan % Feb % Mar % Apr % May % Jun % Jul % Aug % Sep % Oct % Nov % Dec % Jan % Feb % Mar % Apr % May % Jun % Jul % Aug % Sep % Oct % Nov % Dec % Fiscal Year Calendar Year Fiscal Year Calendar Year Table 5-4 shows the application of a joint savings adjustment for upstream EE and the percent savings in electric consumption per home for the Expansion cohort. Seattle City Light Home Energy Report Program Impact Evaluation 38

44 Table 5-4: Expansion Cohort Impact Estimates with Adjustment for Dual Participation Date Range Total kwh Savings in Treated Homes Savings Attributable to Other Programs Net HER Impact Estimate kwh Control Group Usage (kwh) Percent Savings Fiscal Year 2014 Calendar Year 2014 Fiscal Year 2015 Calendar Year , % , % , % , % CPW Legacy Cohort The average customer in the CPW Legacy cohort who received treatment saved kwh during calendar year 2014, which equates to 5.0% of total consumption. However, 2015 estimated savings were significantly lower at 0.61kWh/day (a net per-home savings of kwh per year), or a 1.95% reduction. Estimated impacts displayed significant variability throughout the year, with no clear pattern of seasonality. Table 5-5 shows the average daily savings per home on a monthly basis for the CPW Legacy cohort. Table 5-5: CPW Legacy Cohort Impact Estimates by Month Date Average Customer Impact (kwh/day) Control Group Usage (kwh/day) Percent Load Impact Oct % Nov % Dec % Jan % Feb % Mar % Apr % May % Jun % Jul % Aug % Sep % Oct % Nov % Dec % Jan % Feb % Mar % Seattle City Light Home Energy Report Program Impact Evaluation 39

45 Date Average Customer Impact (kwh/day) Control Group Usage (kwh/day) Percent Load Impact Apr % May % Jun % Jul % Aug % Sep % Oct % Nov % Dec % Fiscal Year Calendar Year Fiscal Year Calendar Year Table 5-6 shows the application of a joint savings adjustment for upstream EE and the percent savings in electric consumption per home for the CPW Legacy cohort. Table 5-6: CPW Legacy Cohort Impact Estimates with Adjustment for Dual Participation Date Range Fiscal Year 2014 Calendar Year 2014 Fiscal Year 2015 Calendar Year 2015 Total kwh Savings in Treated Homes Savings Attributable to Other Programs Net HER Impact Estimate kwh Control Group Usage (kwh) Percent Savings , % , % , % , % It is extremely important to keep in mind that, while the impact estimates for this cohort seem much higher than for the Legacy and Expansion cohorts both on a percentage and absolute basis these results are measured against an extremely small control group due to the reassignment of many of the original CPW area control group customers to treatment during the 2011 expansion of the program. As such, many of the point estimates lack statistical significance due to the extremely large confidence intervals, which will be discussed in further detail in Section 5.2. Given the small sample size for the control group it is not really even possible to conclude that customers exposed to this treatment experienced a statistically Seattle City Light Home Energy Report Program Impact Evaluation 40

46 significant reduction in electricity consumption much less that their energy savings were greater than that of the Legacy or Expanded waves CPW Expansion Cohort The average customer in the CPW Expansion cohort who received treatment saved kwh during calendar year 2014, which equates to 4.8% of total consumption. Impacts in 2015 were lower as well, with 0.94 kwh/day (or a net savings of 344.5kWh/customer) or 3.8% savings. Estimated impacts showed some variability on a percentage basis, but no clear pattern of seasonality. These absolute impact results (kwh) are in line with what was found for the Legacy and Expansion cohorts. However, the CPW Expansion cohort s percent savings estimates are a full 1% to 2% higher than either the Legacy or Expansion cohort because of the low average annual electric usage of the assigned homes (~9,500 kwh annually). Table 5-7 shows the average daily savings per home on a monthly basis for the CPW Expansion cohort. Table 5-7: CPW Expansion Cohort Impact Estimates by Month Date Average Customer Impact (kwh/day) Control Group Usage (kwh/day) Percent Load Impact Oct % Nov % Dec % Jan % Feb % Mar % Apr % May % Jun % Jul % Aug % Sep % Oct % Nov % Dec % Jan % Feb % Mar % Apr % May % Jun % Jul % Aug % Sep % Seattle City Light Home Energy Report Program Impact Evaluation 41

47 Date Average Customer Impact (kwh/day) Control Group Usage (kwh/day) Percent Load Impact Oct % Nov % Dec % Fiscal Year Calendar Year Fiscal Year Calendar Year Table 5-8 shows the application of a joint savings adjustment for upstream EE and the percent savings in electric consumption per home for the CPW Expansion cohort. Table 5-8: CPW Expansion Cohort Impact Estimates with Adjustment for Dual Participation Date Range Total kwh Savings in Treated Homes Savings Attributable to Other Programs Net HER Impact Estimate kwh Control Group Usage (kwh) Percent Savings Fiscal Year 2014 Calendar Year 2014 Fiscal Year 2015 Calendar Year , % , % , % , % 5.2 Precision of Findings Figure 5-1, Figure 5-2, Figure 5-3, and Figure 5-4 show the point estimates for the average load impact (daily kwh savings) for assigned customers of each cohort (blue markers), as well as the 95% confidence interval (green markers). Confidence intervals that encompass the x-axis indicate that the accompanying point estimates are not statistically significant, as was explained in more detail in Section 4.3. The Legacy and Expansion cohorts have significant load impact estimates throughout the evaluation period. These cohorts have large samples, and the observed impacts are sufficiently large to be estimated with confidence. The CPW area cohorts, on the other hand, are either not significant or marginally significant for many months during the evaluation period. The CPW Legacy cohort in particular shows extremely large confidence intervals due to its small control Seattle City Light Home Energy Report Program Impact Evaluation 42

48 group size. Even though the point estimates for the load impacts for that cohort are very large, they are questionable due to the lack of statistical significance. Figure 5-1: 95% Confidence Intervals Associated with Legacy Cohort Impact Estimates Seattle City Light Home Energy Report Program Impact Evaluation 43

49 Figure 5-2: 95% Confidence Intervals Associated with Expansion Cohort Impact Estimates Of particular interest for these results is the seasonal trend in impacts. Across the Legacy and Expansion waves, impacts clearly peak in winter months. Customers in the CPW Legacy and CPW Expansion groups exhibit some evidence of seasonality, but due to their small control group sizes and corresponding lack of precision, it is hard to say definitively that these customers also experience seasonal impacts. Seattle City Light Home Energy Report Program Impact Evaluation 44

50 Figure 5-3: 95% Confidence Intervals Associated with CPW Legacy Cohort Impact Estimates Figure 5-4: 95% Confidence Intervals Associated with CPW Expansion Cohort Impact Estimates Seattle City Light Home Energy Report Program Impact Evaluation 45