FY2010. Energy Efficiency Portfolio. Evaluation Summary. Prepared by: The SRP Measurement & Evaluation Department and The Cadmus Group, Inc.

Transcription

1 FY2010 Energy Efficiency Portfolio Evaluation Summary Prepared by: The SRP Measurement & Evaluation Department and The Cadmus Group, Inc. August, 2010

2 INTRODUCTION AND PURPOSE... 3 SRP s Sustainable Portfolio... 3 Energy Efficiency and Demand Response... 5 Components of a Standard Program Evaluation... 6 FY10 SRP EE PORTFOLIO SAVINGS FY10 SRP ENERGY EFFICIENCY PROGRAMS Residential Programs-Findings and Recommendations PowerWise Appliance Recycling...15 Appliance Rebate...16 PowerWise Cool Cash Rebate...17 HVAC Check-up...20 PowerWise Homes...20 Solar Shade Screens...22 Home Checkup Pilot...23 CFL Distribution Program...24 MPower...26 Low-Income Weatherization...26 PowerWise Retail Education...28 Low Income Education...28 Commercial Programs-Findings and Recommendations PowerWise Standard Business...31 PowerWise Custom Business...33 PowerWise Large Business...35 PowerWise Compressed Air...37 PowerWise Retro-commissioning...39 Commercial New Construction...41 SPATIA Energy Information System...42 Demand Response...44 CONCLUSION SRP FY10 Energy Efficiency Program: Evaluation Summary 2

3 INTRODUCTION AND PURPOSE This document is divided into two sections. First is an overview section which is intended to provide a backdrop and summary of the purpose and methodology for this evaluation effort. It includes a background description of SRP s Energy Efficiency initiatives, the components of energy efficiency program evaluations, and high-level findings including program impacts, costs, participation levels, and cost effectiveness. The following section provides a detailed program-by-program review of evaluation findings for SRP s energy efficiency programs in effect during fiscal year 2010 (FY10), which ranged from May 1, 2009 April 30, SRP s Sustainable Portfolio In FY04, SRP s Board of Directors adopted the Sustainable Portfolio Principles, which set a target for sustainable resources inclusive of both renewable energy and energy efficiency measures. In FY06, the SRP Board approved a new target of 15% of retail energy requirements by 2025 from sustainable resources. The current plan for how to reach this target is shown in Chart 1. Once the new standard was adopted, SRP commissioned an outside consultant, Quantec(now the Cadmus Group), to conduct an energy efficiency potential study to determine what programs SRP could cost effectively grow in its service territory. This study was completed in FY08. In FY09, SRP began to implement the most cost effective of these measures and has a plan to steadily increase its investment in these programs over its six-year financial planning horizon. Measurement, Validation, and Evaluation of these energy efficiency programs is conducted in accordance with prevailing industry standards. SRP FY10 Energy Efficiency Program: Evaluation Summary 3

4 Percent of Retail Requirements Chart 1-Sustainable Portfolio Process (SPP) Resource Targets 20% 15% 10% SRP Sustainable Portfolio Chart *Renewable and energy efficiency resources in excess of requirement will be banked for future use SRP SPP REQUIREMENT RENEWABLE RESOURCES 5% 0% HYDRO RESOURCES CONSERVATION - Energy Efficiency and Demand Response As a result of the inclusion of specific targets, SRP includes these resources in its overall resource plan. With additional program experience and savings validation through this measurement and evaluation process, SRP will be able to more directly compare energy efficiency resources with other traditional and renewable supply side options. SRP s ongoing program evaluation builds sufficient confidence in the energy (kwh) savings from these programs to allow them to be relied upon as a system planning tool in SRP s Integrated Resource Planning process. SRP FY10 Energy Efficiency Program: Evaluation Summary 4

5 Energy Efficiency and Demand Response Energy Efficiency (EE) and Demand Response (DR) programs target customer behavior and investment as a means by which electric utility companies can help meet consumer demand. It is broadly seen as an alternative to traditional supply-side resources in meeting new customer demand. EE and DR programs are currently experiencing resurgence across the nation. Utilities invest in these programs for a variety of reasons, including: Regulatory Requirement o Many states and regulatory commissions have imposed mandatory standards o Federal standards being developed allow a portion of renewable energy standards to be met by EE Risk Management o Future demand and supply estimates include an element of risk; investing in EE serves as a hedge by broadening the portfolio of resources available to help utilities meet future customer demand for electricity Emission Avoidance o o Every unit of energy saved is a unit of pollution avoided This point has been a key element of government policy as a means by which to help mitigate global warming concerns Customer Satisfaction o o o Many customers value energy efficiency tools and programs as a way to help them control their energy costs. This is especially the case in times of economic hardships. Access to utility real-time information dynamic pricing structures used for product pricing promote understanding of the relationship between cost and electricity use for retail customers EE provides a means by which consumers can take action to reduce their energy consumption and lower their electric bills Generation Benefits o Investing in EE can result in deferral of capacity and energy investments, lower utility fuel costs, and reduce reserve requirements Possible Transmission and Distribution Benefits o o When EE reaches a critical mass, it may also allow utilities to defer new investment in transmission and distribution Demand side programs can also improve reliability, especially near real-time demand response programs SRP FY10 Energy Efficiency Program: Evaluation Summary 5

6 Research Objectives In general, the long-term research objectives for SRP s EE program evaluations will include the following: a review of secondary literature to determine how the industry is evaluating similar programs an assessment of assumptions used to calculate and confirm kw and kwh savings attributable to program activity an evaluation of program processes, education, and marketing materials an assessment of customer experiences and satisfaction a determination of market saturation and transformation Sometimes determining answers to these questions requires a multi-year review of a program s performance. SRP plans to provide written annual updates on progress in each of these areas, as appropriate. Benefits Accrue Differently to Stakeholders When energy efficiency is sponsored by the utility, it can raise questions concerning equity since under most circumstances it leads to an increase in customer prices. Avoided-cost analysis has been widely used in the energy sector to assess the cost-effectiveness (or net benefits) of EE relative to conventional supply alternatives. When calculating benefits, analysts begin with avoided costs and make adjustments for administrative and direct program costs as well as other costs associated with participating in the programs. Historically, investment decisions in EE were made on a measure-by-measure basis. Increasingly, EE professionals recognize the importance of a portfolio strategy for design and evaluation. If each measure included in the program must be deemed cost-effective on its own, the diversification benefits and economies of joint delivery of the bundled program are ignored. A portfolio approach analyzes combinations of measures to determine the most cost-effective program design and provides additional risk mitigation through portfolio management. Components of a Standard Program Evaluation The general approach to EE program evaluation includes a review of the following factors: Process Evaluation o Review of program administration and marketing efforts as well as customer and stakeholder satisfaction, with the goal of finding program improvement opportunities Impact Evaluation o Review of whether the expected kwh savings, or avoided energy consumption, was realized SRP FY10 Energy Efficiency Program: Evaluation Summary 6

7 Cost-effectiveness Evaluation o By using standard cost benefit tests, and looking at the short-term and lifetime cash flow implications of a program, resources can be aligned with the best performing programs Of course, all of these objectives must be managed in the context of aligning the program evaluation cost with total program costs. Typically, evaluation costs fall within 5% to 10% of total program costs. For this reason, varying levels of rigor were applied across SRP s FY10 program evaluation efforts, depending on the relative size and potential longevity of the programs, and how much is already known about them. Process Evaluation: The Customer Experience Process evaluation includes a review of the program design, administration, implementation and delivery, and market response. Interviews with key program and marketing staff are conducted to determine process elements and potential areas for improvement. Additionally, customer surveys are conducted with both program participants and non-participants to understand the strengths and weaknesses of the program processess from the customer perspective as well as the customer s response to the various program elements. Information from all these sources are evaluated to identify potential improvements, not only to increase the cost effectiveness of the program, but also to improve the likelihood of its acceptance by the customer. Impact Evaluation There are several ways to approach impact evaluation, or energy savings calculations, for energy efficiency programs. These include conducting a billing analysis of weather-adjusted customer consumption before and after a measure is installed, directly measuring consumption of the replaced technology before and after, use of regression analysis of expected savings based on a formulaic representation of the measure and statistical relationships derived from mathematical calculations, or by applying an engineering model which defines the expected consumption delta between the replaced technology and the newly installed technology for any given application. When the unit savings for a given measure is small relative to total household consumption, the analysis is best handled with an engineering model approach that generates a deemed savings per unit, rather than using direct measurement or billing analysis. This is more practical and far less costly to administer for programs with wide reach across the customer base. Because a key goal of EE programs is to reduce the need for investment in new supply side resources, a critical part of the impact evaluation is determining the amount of generation avoided by the implementation of the program. This is accomplished by grossing up the program level savings to represent them at the generation level. Program savings are determined at the customer meter level and transmission and distribution line losses need to be added to these numbers to determine the amount of generation avoided by the program. T&D losses vary by season, time of day and customer class and are applied to the various program elements appropriately. SRP FY10 Energy Efficiency Program: Evaluation Summary 7

8 Another key component of the impact evaluation is the valuation of adjustments to the gross, or maximum, program savings. The most common adjustment is the Net-To-Gross ratio, which adjusts for freeridership on the program. Freeridership represents consumers that would have engaged in a given program activity (such as the purchase of CFLs) without the existence of the utility (or other market actor) incentive. Since programs are implemented to incent customers to take action they would not have taken on their own, the savings associated with free rider participation cannot be attributed to program influence. To assess the true value and impact of the program, freeridership must be taken into account. Determining what the exact impact is of all these factors is an art as much as it is a science. Utilities and other market actors can spend a great deal of time and money trying to nail down the exact impact of programs by a variety of measures. It is important to remember that such evaluation expenditures must be in line with total program size relative to the overall portfolio. The valuation of these factors is most commonly derived through direct customer inquiry, conducted as part of the process evaluation. Cost Effectiveness Evaluation While maximizing energy efficiency impact is the primary goal of any EE program, resources available to develop and administer initiatives are limited. Therefore, utilities consider the relative cost effectiveness of programs. This is generally determined by comparing the costs required to run the program with the benefits delivered. If the ratio of benefits to costs exceeds 1, the program is determined to be cost effective. There are two main categories of benefits, or avoided costs: energy-related and capacity-related. Energy-related avoided costs refer to market prices of energy, fuel costs, natural gas commodity prices, and other variable costs. Capacity-related avoided costs refer to infrastructure investments such as power plants, transmission and distribution lines, and pipelines. From an environmental point of view, saving energy reduces air emissions, including greenhouse gases (GHGs). Within each of these categories, policy-makers, program developers, and evaluators must decide which specific benefits are sufficiently known and quantifiable to be included in the cost-effectiveness evaluation. The choice of where to apply each cost-effectiveness test has a significant impact on the ultimate set of measures offered to customers. In general, there are three places to evaluate the cost-effectiveness test: at the measure level, the program level, and the portfolio level. Applying cost-effectiveness tests at the program or portfolio levels allows some non-cost-effective measures or programs to be offered as long as their shortfall is more than offset by cost-effective measures and programs. 1 1 National Action Plan for Energy Efficiency (2008). Understanding Cost-Effectiveness of Energy Efficiency Programs: Best Practices, Technical Methods, and Emerging Issues for Policy-Makers. Energy and Environmental Economics, Inc. and Regulatory Assistance Project. SRP FY10 Energy Efficiency Program: Evaluation Summary 8

9 PortfolioPro SRP employs the Portfolio Pro software tool to evaluate the cost effectiveness of its energy efficiency programs both on the individual and portfolio level. Developed by the Cadmus Group, Portfolio Pro is a cost effectiveness model that allows utilities to ascertain program cost effectiveness from multiple perspectives. It provides a link between the results of conservation and demand response potential assessment and program development and allows users to create programs from individual measures and combine them into portfolios. A database of common assumptions such as financial inputs, hourly load shapes and avoided costs also allows evaluators to assess resource valuation and risk assessment across all EE programs. Cost Effectiveness Tests There are different perspectives from which the measurements of costs and benefits can occur. EE programs that provide rebates for energy efficient equipment purchase and installation, for example, have costs associated with the payment of these rebates from the perspective of the utility. To the utility customer participating in the program, however, the rebate is a benefit. This differing realization of program costs and benefits yields a number of differest cost/benefit tests, each capturing a measure of cost effectiveness from a different perspective. SRP evaluates economic performance using two tests: Total Resource Cost (TRC) Rate-Impact Measure (RIM) SRP uses the TRC to inform decisions on measure, program and portfolio design. SRP also uses the RIM test to ensure programs do not unduly impact electricity pricing. Benefit-cost ratios are calculated using the methods described in the California standard protocols for analyzing cost-effectiveness of conservation programs. 2 2 California Standard Practice Manual for Economic Analysis of Demand-Side Management Programs and Projects, California Energy Commission, October SRP FY10 Energy Efficiency Program: Evaluation Summary 9

10 Table 1 details the cost effectiveness tests used by SRP in EE program evaluation. Table 1-Summary of Benefits and Costs Included in Each Cost-Effectiveness Test Total Resource Cost (TRC) Test Benefits Costs Rate-Impact Measure (RIM) Energy-related costs avoided by the utility Capacity-related costs avoided by the utility Additional resource savings Monetized environmental and non-energy benefits Energy-related costs avoided by the utility Capacity-related costs avoided by the utility Lost revenue due to reduced energy bills Program overhead costs Program installation costs Incremental measure costs Program overhead costs Utility/program administrator incentive costs Utility/program administrator installation costs Source: California Standard Practice Manual: Economic Analysis of Demand-Side Programs and Projects. SRP FY10 Energy Efficiency Program: Evaluation Summary 10

11 The Future is Uncertain It is a commonly held prudent business practice to hedge risk in order to mitigate uncertainty. The electric utility industry faces much uncertainty today, including increases in the cost of new supply additions, high fuel and market price volatility, uncertain future demand, uncertain carbon mitigation costs, and a long capital planning process that makes management of such uncertainties critical. Regulatory mandates, and carbon regulation in particular, expose utilities to significant emissions risk. Other future challenges, such as increased electrification of household and commercial/industrial functions and the adoption of electric vehicles, will further change the operating landscape for electric utility companies. Historically, Arizona has also needed to manage the demand of a rising population and rising per capita consumption. Now the future effect of population and per capita consumption is unclear. All of these factors, and more, must be managed under the backdrop of an obligation to serve. Hedging risk mitigates price risk to the company and to its customers. SRP takes many actions today to mitigate risk, including: prudent planning processes; long planning horizons; investing in a portfolio of resources, including renewable energy and EE; conducting research for future supply and demand side technologies; fuel hedging programs; and more. EE programs are one tool, an important tool, by which utilities can manage risk and create a portfolio of resources that will allow SRP to continue to serve the needs of its customer base for the future, whatever it may hold. Energy Efficiency programs can also serve to cost-effectively reduce risk by reducing exposure, reducing volatility, diversifying supply options, and creating new real-time resources such as demand response programs. SRP is actively pursuing the expansion of these options. However, SRP needs to also be prudent in the expansion of, and reliance upon, demand side resources (and other new resources) to ensure that resource can deliver the assumed benefits avoided consumption and life expectancy, or persistence, of these savings. This is the primary purpose of the MV&E function to help build confidence in these resources by the systematic validation of assumptions used in program design and development. These findings are then applied back into the development and implementation cycle to improve the process and provide customers with a better overall product and ensure wise investments in meeting forecasted increases in electricity consumption. SRP FY10 Energy Efficiency Program: Evaluation Summary 11

12 FY10 SRP EE PORTFOLIO SAVINGS Table 2 presents cost-effectiveness results at the program and portfolio level for SRPs FY10 suite of programs. Table 2-FY10 Cost-Effectiveness Results Residential Programs TRC RIM Appliance Recycling Appliance Rebate CFL Distribution Cool Cash Rebate HVAC Checkup Sunscreens PW Checkup Pilot PowerWise Homes Low-Income Weatherization Low-Income Education M-Power Commercial Programs TRC RIM Standard Business Custom Business Large Business Compressed Air Retrocommissioning Demand Response* SPATIA EIS Residential Portfolio Commercial Portfolio* Total Portfolio* *Demand Response not included in Portfolio-level results SRP FY10 Energy Efficiency Program: Evaluation Summary 12

13 The following tables provide a summary of SRP s FY10 EE program savings. Additional information regarding each of the listed programs is provided in subsequent sections of this document. Table 3-Summary of FY10 Residential Program Results Program Number of Participants Ex Ante[1] Net Evaluated Gross Ex Post[2] Program Savings Evaluated Net-to- Gross[3] Evaluated Net Ex Post[4] Program Savings Net Realization Rate[5] Program Life Lifetime Savings Net Coincident Peak (MWh) (MWh) (MWh) (MWh) (MW) RESIDENTIAL PROGRAMS Appliance Recycling 10,018 11,443 11, , , Appliance Rebate 14, , , , Cool Cash HVAC Checkup Sun Screens [6] Home Checkup Pilot [6] PowerWise Homes M-Power Low Income Weatherization Low Income Education CFL Sub-Total 2,140 3,719 2, , , ,824 N/A N/A 15 7, N/A N/A ,966 3,109 4, , , , , , , , , , ,634,925 31,939 77, , , , , , , [1] Planned savings for program year, net of freeridership [2] Evaluated actual program savings before adjustment for freeridership [3] Proportion of savings attributable to program influence (nets out free riders) [4] Evaluated actual program savings adjusted for freeridership [5] Ratio of actual vs. planned savings [6] Programs added midyear and therefore not part of original budget SRP FY10 Energy Efficiency Program: Evaluation Summary 13

14 Table 4-Summary of FY10 Commercial Program Results Program Number of Participants COMMERCIAL PROGRAMS Ex Ante[1] Net Evaluated Gross Ex Post[2] Program Savings Evaluated Net-to- Gross[3] Evaluated Net Ex Post[4] Program Savings Net Realization Rate[5] Program Life Lifetime Savings Net Coincident Peak (MWh) (MWh) (MWh) (MWh) (MW) Standard Business ,902 73, , , Custom Business 92 15,126 10, , , Large Business 2 2, , Compressed Air Solutions 3 5,144 3, , , New Construction Retro- Commissioning 1 1,669 1, , , Demand Response* Spatia 77 3,548 1, , , Sub-Total* 56,110 91,610 65, , TOTAL* 257, , ,395 1,521, * Demand Response not included in Portfolio-level results [1] Planned savings for program year net of freeridership [2] Evaluated actual program savings before adjustment for freeridership [3] Proportion of savings attributable to program influence (nets out free riders) [4] Evaluated actual program savings adjusted for freeridership [5] Ratio of actual vs. planned savings SRP FY10 Energy Efficiency Program: Evaluation Summary 14

15 FY10 SRP ENERGY EFFICIENCY PROGRAMS Residential Programs-Findings and Recommendations PowerWise Appliance Recycling Process Findings SRP increased this program s marketing budget and assigned dedicated marketing management. The increased investment and focused promotion of the program resulted in increases in units recycled each month, more balanced participation, and an increase in program awareness. About half the nonparticipants who recycled units outside of the program were not aware of the program. The remainder stated they did not have enough information to participate. The current program design excludes small landlords, as rebates can only be paid to account holders for the addresses where appliances are located. The launch of the Arizona Public Service (APS) appliance recycling program will benefit the SRP program, especially as it increases the total marketing effort in the shared media markets. The implementer improved its process for collecting configuration data that were often missing in FY 2009 and recommended for collection. Process Recommendations Marketing efforts should be continued to raise awareness of the program and provide information about the program. Based on survey data, the marketing team should define strategy and tactics for communicating program information to nonparticipants. If feasible, marketing efforts should coordinated with APS. There could be significant benefits to coordination in terms of leveraging media buys and providing consistent messaging. It is suggested SRP explore revision of the payment policy to enable expansion of the program to small landlords who are currently excluded because rebates must be paid directly to account holders who generally are the tenants. This policy change has the potential to increase program unit volume and savings with little or no incremental costs. Impact Evaluation Findings The Appliance Recycling program paid rebates for the destruction of 10,016 old refrigerators and freezers in Fiscal Year 2010, achieving 95% of the targeted participation level of 10,500 units. The sample of 70 participants and 70 nonparticipants surveyed for this evaluation required the NTG ratio and part-use factor to be calculated at the program level. The percentage of appliances found not to be in use (8.6%) was substantially lower than that found in the FY09 evaluation, especially for freezers. SRP FY10 Energy Efficiency Program: Evaluation Summary 15

16 Per-unit savings for freezers increased considerably compared to that found in the FY09 evaluation, and per-unit savings for refrigerators decreased very slightly. The program s 34% freeridership was found to be closely comparable to freeridership rate found in the FY09 evaluation (33%). Lower-than-planned per-unit gross savings drove the program s low realization rate. However, the per-unit gross savings found in this evaluation were comparable to those found in the FY09 evaluation as well as to gross savings estimates for similar programs run by other utilities. Changes in the part-use factor were typical of a second-year program, during which fewer unused units are recycled. Although the percentage of partially used and unused appliances was still rather high, Cadmus previous experience with maturation of appliance recycling programs suggests the rate will continue to decrease as the program matures. These findings resulted in evaluated net savings of 7,663,955 kwh, approximately 33% lower than anticipated program savings. Impact Evaluation Recommendations Cadmus recommends SRP consider increasing the sample size for participant and nonparticipant surveys to allow for stratification by measure. A larger sample size would allow more robust results, as specific NTG ratios and part-use factors could be calculated for refrigerators and freezers separately. This would provide a more detailed examination of these two drivers of program impact, and might provide valuable insights that could influence future program planning. Appliance Rebate Process Findings and Recommendations No process evaluation was done in FY10. The program has not proven to be cost effective from an energy efficiency standpoint but is offered for it s societal benefits (primarily water savings). Impact Evaluation Findings In FY10, the program exceeded the original volume target of 3,118, with a full-year total of 14,572, or more than 4.5 times the original target. Gross savings per unit were generally consistent with planning assumptions. Unit savings for Tier 2 dishwashers were the exception, which neared the planning assumption for the first 8 months of the year, but then declined as new standards took effect. High freeridership limited net savings for appliances rebated in FY10. Freeridership was determined to be 60%, which was much higher than the 20% used for planning purposes. This SRP FY10 Energy Efficiency Program: Evaluation Summary 16

17 was in large part due to the introduction of Federal rebates which heavily discounted program savings and was one of the major factors leading to the program not being cost-effective. Analysis of the participant database found 6% to 8% of units did not have complete information on fuel types (for water heaters and clothes dryers). Savings from these units were counted with the fuel type data assumed, based on the mix of the known units. The program worked well as designed, processing over four times the anticipated number of rebates. However, it was not cost-effective. For dishwashers, the new minimum efficiency level required by federal statute limits the savings per unit even for the highest efficiency level (CEE Tier 2). Savings are higher for homes with electric water heat (60 kwh per year) than for homes with gas water heat (12 kwh per year). The federal standard also limits water use for all dishwashers; so water savings are fairly modest, at 484 gallons per year. Since savings are limited and freeridership high, the value of a rebate on dishwashers is low relative to other measures. For clothes washers, savings of water and electricity are substantial for any of the three CEE Tiers. Electric savings are twice as high for homes with electric water heat than for homes with gas water heat. Impact Evaluation Recommendations Cadmus recognizes there are many reasons to operate programs that are not cost-effective (TRC less than one). SRP could consider targeting appliances in the highest-efficiency tier for rebate, and limiting participation to homes with electric water heaters. SRP could consider eliminating dishwasher rebates; electricity and water savings are both very modest. If the program continues to be offered, Cadmus recommends implementing improvements to data collection process(es) to provide a complete data set for energy savings calculations. PowerWise Cool Cash Rebate Process Findings The Cool Cash program successfully encouraged customers to install more efficient equipment than required for the federal tax credit. Units rebated were more efficient than those assumed during program planning. Approximately 60% of participants stated they purchased a more efficient cooling unit than originally planned because of the Cool Cash program. The program goals and rebate requirements are appropriate, given the manufacturer s rebates and federal tax credits already available to SRP customers. Customer satisfaction with the program is high, and customers have responded well to SRP s marketing efforts. The larger proportions of the customers who have not heard of the program describe themselves as Hispanic. SRP FY10 Energy Efficiency Program: Evaluation Summary 17

18 Contractors remain positive about the program, but remain confused about program equipment efficiency requirements. In addition, their request that SRP lower efficiency requirements to match the tax credit or dovetail with the APS program indicates they need clarifying information from SRP. Process Recommendations Continue to take advantage of the momentum provided by the tax credit and manufacturer rebates. Maintain current customer and eligibility requirements while the tax credit remains available. SRP should collect information on program nonparticipants from its existing billing and marketing databases. SRP could include the Cool Cash program in all future Hispanic marketing plans. In addition, SRP may want to investigate whether customers who had not heard of the program shared some uniform characteristics such as e-bills or the managed payment plan and, if necessary, develop a different marketing approach to reach these customers. Provide additional information to contractors so that they understand program equipment eligibility requirements. Strengthen the partnership with contractors and offer additional information resources: provide regular informational meetings; offer a dedicated web site; set up a contractor telephone hot line. Develop an on-line application rebate system for contractors. Impact Evaluation Findings The Cool Cash program rebated 2,140 cooling units in FY10. Actual program rebates represented 67% of the SRP program planning assumptions, which estimated 3,187 participants. Evaluated gross savings achieved for the Cool Cash program were 2,200,000 kwh or 76% of SRP s planned estimated savings of 2,904,000 kwh. Evaluated freeridership was 17%, which was lower than that used in SRP s program planning assumptions (20%). The SRP M&E reporting assumption for freeridership in this program was 8%. This difference in evaluated freeridership, along with a lower-than-anticipated participation rate, yielded net savings at 69% of original planned savings. Actual FY10 savings provided in the SRP FY10 final program reporting Dashboard calculated a savings estimate for each individual HVAC unit rebated under the program. It subtracted the annual usage derived using the capacity and SEER of the new unit installed from the annual usage expected from a SEER 13 unit of the same capacity. Cadmus aggregated these by the program s SEER strata (15, 16, 17, 18+), and estimated average savings per stratum. Original program planning estimates of per unit savings for each stratum were derived from an APS study of consumption that seemed uncharacteristically high. The estimates derived using SRP s calculation method (as shown in the Dashboard) were much closer to those Cadmus estimated using Energy-10 modeling. The estimation method used by SRP (as indicated in the Dashboard) to calculate actual savings for the FY10 program year only estimated savings from cooling. However, approximately 75% of the units installed in the FY10 program were heat pumps. SRP FY10 Energy Efficiency Program: Evaluation Summary 18

19 Rebate dollars per month, calculated using program tracking data, did not match rebate dollars in the SRP financial tracking file. However, this appeared to result from a slight time lag between the program s installation date and the actual rebate payment recorded. Evaluation results indicate the program is functioning as planned. SRP customers are motivated to install more efficient equipment than required to qualify for the Federal tax credit. Although heating savings may be small in comparison to cooling savings for the SRP territory, it is worth noting there may be additional heating savings that can be claimed due to the program. In FY09, SRP modified the application form to collect additional information about equipment replaced. These data provided the necessary information to estimate the effective useful life of replaced equipment, helping to determine whether it is appropriate to claim additional savings due to early retirement. Although data collected from FY10 participants indicated the age of removed equipment did not warrant claiming savings due to early retirement, this may not be the case in future years. Additional reporting mechanisms, which may help in future planning and evaluations, include: o o Calculating an average savings per stratum using the savings estimation method used in the SRP Dashboard to calculate savings per SEER stratum for use in future program planning; Implementing a quarterly system to report program participation and savings estimates to allow coordination of participant counting and savings estimation methodologies Impact Evaluation Recommendations To continue leveraging the sales momentum achieved by the addition of the Federal tax credit, Cadmus recommends SRP continue to offer rebates for installation of high-efficiency cooling equipment under the current rebate structure. Cadmus recommends SRP examine available savings that may be realized through calculation of heating savings from Cool Cash heat pump installations. This would likely produce additional savings that could be added to Cool Cash Rebate program estimates for future planning. This could be done if planning estimates currently allocated by SEER are also split out by equipment types (air conditioners and heat pumps). Cadmus recommends SRP continue to collect information about the age of the equipment replaced under the Cool Cash program. Cadmus recommends reviewing summaries of program participation and estimated savings on a quarterly basis so any inconsistencies can be dealt with early in the evaluation process. Cadmus recommends that SRP calculate average savings per SEER stratum using the savings estimation method used in the SRP Dashboard for use in future program planning. This method subtracted the annual usage derived using the capacity and SEER of the new unit installed from the annual usage expected from a SEER 13 unit of the same capacity. Aggregating the resulting savings values by the program s SEER strata (15, 16, 17, 18+) allows for calculation of average savings per SEER stratum. Estimates developed using participant data from prior program years SRP FY10 Energy Efficiency Program: Evaluation Summary 19

20 can be used in the development of program planning estimates of savings per SEER for subsequent program years. HVAC Check-up Process Findings The Program was generally viewed as very successful for both participants and contractors The administration of the Program by the Electric League of Arizona (ELA) is very good - mechanisms for certifying contractors, promoting the program, fielding customer calls, referring contractors, fielding any complaints, and arbitrating any disputes all function extremely well. Customer complaints noted in FY09 have not appeared again in FY10 The ELA phone system upgrade should handle expanded spring 2010 offering without any problems. Expanded marketing is planned to coincide with Spring 2010 buy-down including print ads, web banners, and promotion at retail events. Contractors pushed for a price increase (to $64.95) that did not get approved, but will likely be a part of the FY11 program. Process Recommendations Emphasize to contractors that customer s expectations need to be well informed prior to the visit Consider merging the fall and spring applications into one document. Include on the data collection form whether a Cool Cash referral took place. Add a data field to the tracking database that captures how the customer heard of the program. Integrate TrackSmart database into program process. Impact Evaluation No Impact Evaluation was done. PowerWise Homes Process Findings Largely due to the building slowdown, program KAMs have experienced an increase in builderinitiated contract changes. This has increased KAM administrative workloads associated with processing these changes and leaves KAMs with less time to sell the program to builders or train SRP FY10 Energy Efficiency Program: Evaluation Summary 20

21 other program actors, such as real estate agents and appraisers. Ultimately, this may translate to fewer participating builders and fewer certified homes. Contract batches that are significantly smaller than a community of homes has resulted in oversampling PowerWise Homes for inspection. Such oversampling creates an excessive amount of field work for inspectors and unnecessary amounts of administrative paperwork for program staff. SRP sampling protocol for inspections are more rigorous than the RESNET protocol used in other residential new construction programs. Using RESNET protocols would not likely adversely impact home quality or program savings. Therefore, unless a demonstrated need occurs in the future for higher sampling rates, such as frequent failures identified through testing or inspection, this sampling level seems unnecessary. The measures added in January 2010 largely fulfill FY 2009 recommendation. Inclusion of these measures could allow SRP to determine where and how quickly the market implements higher levels of system efficiency and building practices. Program staff has substantially implemented process follow-up and data collection recommendations from the 2009 evaluation. No data elements in addition to those being collected were identified. Process Recommendations Consider ways to reduce program administrative workloads. These could include shifting some KAM clerical work to other staff to free up KAMs time to spend with builders and other market actors impacting the program s performance. This would directly address constraints on KAM resources, but would not unburden the entire contracting system. Consider reducing workload by increasing the numbers of homes contracted in a single batch. Larger batches would significantly diminish the number of contract and contract modification transactions handled throughout the program, at no additional cost. If PowerWise homes continue to be high quality, and tests identify few failures, SRP may want to relax inspection requirements. SRP should carefully track the measures installed that meet requirements for additional incentives. Tracking measures used by participating builders to reach higher levels of overall home performance. This will allow program staff to know when a new performance tier is needed. Consider adding a third high efficiency tier to capture high performing homes in one contract batch. Impact Evaluation Findings In FY10, 3,966 homes were certified under the Power Wise Homes program. The program exceeded original participant estimates of 1,420 by close to 280%. Evaluated savings achieved SRP FY10 Energy Efficiency Program: Evaluation Summary 21

22 4,858,967 kwh (exceeding the adjusted original estimate of 3,124,077 kwh by approximately 156%), driven by the higher-than-expected participation. Savings estimates by tier from the billing analysis, using FY09 participant data, were substantially lower than those estimated in the FY09 evaluation using participants. Program changes in FY10 provide builders additional incentives for incremental measures. These changes encourage incremental increases in energy efficiency. The lower tier level savings estimates may be due to trends toward smaller homes (square footage) as well as to builders implementing energy-efficient building practices in general. These and other factors resulted in energy consumption of nonparticipant homes moving closer to that of PowerWise homes. Recent program changes are expected to result in increases in energy efficiency and correspondingly larger differences in energy consumption between the PowerWise and non- PowerWise homes. Impact Evaluation Recommendations For the FY11 participant evaluation, Cadmus recommends a new survey with participant and nonparticipant builders be conducted to assess current standard building practices. This will help to inform program offerings, and determine builder responses to recent program changes to increase energy efficiency of PowerWise Homes. Cadmus recommends a billing analysis be included in the FY11 evaluation, comparing FY10 participants and nonparticipants with a full year of post-occupancy consumption data. This will allow SRP to evaluate and update savings estimates. Solar Shade Screens Process Findings The Program is generally very popular with contractors; there is a high degree of willingness on their part to cooperate and meet program requirements. The contractors are generally smaller firms with less technical training and experience than those participating in other programs. Anecdotal evidence suggests that concern may exist on the part of customers regarding lag time between initial call and installation (sometimes 3-4 weeks), however ELA has not fielded any complaints regarding this issue. The deemed annual energy savings estimate was revised from 8.5 kwh/installed ft2 to 5.0 kwh/ installed ft2 The program rebate was reduced from $0.80/ft2 to $0.50ft2. As a result, Ace Hardware ceased participating. There were no reported problems from ELA due to reduction in rebate. SRP FY10 Energy Efficiency Program: Evaluation Summary 22

23 The ELA refined the program rebate process to allow for contractors to receive incentives more quickly Process Recommendations Develop program protocol for installation verification, i.e. inspect a sample of reported installation. There is currently no inspection or verification protocol to ensure that installations completed and submitted for rebates meet the program eligibility requirements. The program design should be revised to include a mechanism for verification of a sample of the installed shade screens. Consider revision of the rebate structure so SRP rebate check goes directly to customer. Currently, the cost of the shade screens to the customer is inclusive of the rebate amount, and the contractor recoups the cost after installation is complete. SRP could possibly benefit from increased visibility by restructuring the rebate process such that the customer receives a check directly from SRP after providing documentation of shade screen installation. Increase available contractors to include more who are bilingual. A significant portion of SRP s residential customer base is Hispanic, and currently the contractors participating in the program have limited bilingual staff, leading to under-delivery of the program to this customer segment. Provide additional education/training and informational resources to contractors. The contractor firms participating in the shade screen program are usually smaller in size and do not employ staff with advanced technical experience. Additional training and information about the benefits of shade screens, and potentially best practices for marketing and advertising could provide valuable skills to contractors and increase program. Impact Evaluation No Impact Evaluation was done. Home Checkup Pilot Process Findings The contractors who participated were generally happy with the program and collected valuable information on the houses audited. There is currently no process by which SRP follows up with customers to determine whether the recommendation made as a result of the audit are implemented there are no quantifiable savings. The attempt to provide an unbiased audit experience via contractors who are not encouraged to provide cost estimates, sell equipment, or offer additional services may decrease the likelihood that repairs or retrofits are made. There was very limited marketing activity at the beginning of the program year. In January, the web page was posted. Promotion of the checkup program occurred at retail events. SRP FY10 Energy Efficiency Program: Evaluation Summary 23

24 All data collection was done by hand on a per case basis. The primary features and functions of Home Checkup Pilot Program will be delivered in FY11 via the Home Performance with EnergyStar Program. Process Recommendations Institute a process by which installation/repair verifications occur to better understand the actual customer savings, and allow for a more accurate determination of program potential. There is currently no mechanism to verify any savings resulting from equipment repair or replacement attributable to the program. Considering the significant cost to SRP for each participating customer, the determination of associated energy savings will be essential if the program is to be considered cost effective. An expanded scope of delivery will require automation of the data collection and reporting processes. The manual data collection process does not allow for meaningful or efficient compiling or reporting of data should participation in the program expand significantly. Considering the scope and value of the information potentially available through the audit process, an automation of the process and incorporation of more robust data storage tools is highly recommended. Consider collecting additional detail during audit. The contractor has the opportunity to collect a significant amount of data on each house audited with fairly small incremental resources needed. Detailed information including construction materials, window number/type, building vintage and square footage, unconditioned space characteristics, and similar detailed can provide SRP with an extremely valuable data resource on its residential housing stock. Evaluate whether contractors should provide more detailed cost estimates to customers along with recommendations, or should be encouraged to offer additional services. Contractors currently do not provide cost estimates or referrals to the homeowners that would allow the recommendations made to be implemented. SRP should consider altering this approach, potentially by batching contractors for each audit to include a pool of potential retrofit/repair providers separate from those who perform the audit. Impact Evaluation No Impact analysis was performed. CFL Distribution Program Process Findings From SRP s and manufacturers perspectives, current program relationships are working well. The program is well-positioned in terms of products, channels, and the market, given SRP s strategy of making discounted CFLs available in most stores where SRP customers purchase CFLs. SRP FY10 Energy Efficiency Program: Evaluation Summary 24

25 The CFL market is not saturated in SRP s service territory, and price points are sufficiently compelling to maintain strong sales through the economic downturn. However, data from the survey, manufacturers, and benchmark analysis show significant potential for future growth. Though SRP significantly increased the program s marketing budget and volume, 20% to 30% of the survey population (depending on the survey group) remained unaware of CFLs or were not using them; and 16% of CFL buyers reported the purchased products were part of the SRP Program. Process Recommendations To maximize growth in the lighting market, the program may want to consider increasing the duration of contracts with manufacturers and retailers in exchange for additional commitments from manufacturers (improved buy-down) and/or retailers (marketing). Additionally, the program should consider adding specialty CFLs, CFL lighting fixtures and LED night lights. SRP should continue to include additional chain stores and smaller stores to grow their sales channels as cost effective. Continue to use the marketing effort to raise awareness among parts of the population unaware of CFLs or not buying them. In addition, the program marketing effort should consider additional indicators of marketing effectiveness to determine if current investments are having an impact or if potential savings can be achieved. Impact Evaluation Findings The program is functioning well, and has delivered strong results in FY10. It successfully sold 1.63 million CFLs in FY10, versus the original participation estimate of 1.05 million units. In terms of volume, the program achieved 155% of its original goal. This led directly to the 166% realization rate for gross evaluated savings. As a planning assumption, NTG was expected to be 69%. Evaluated NTG was somewhat higher at 75%, which was reflected in the 181% realization rate for net savings. Per unit savings were originally assumed to be kwh per year. With no adjustment for the EISA, the per unit savings were found to be slightly higher at kwh per year. The EISA adjustment was found to have a 13.5% impact on savings over seven years. The unit savings, reflecting this projection, are therefore determined to be kwh per year. The EISA adjustment projected in this evaluation was based on the specific assumptions stated. As time passes and the regulations draw nearer, more will be known about the market, and it will be possible to make informed changes to the model and mix of lighting products offered. Impact Evaluation Recommendations The CFL market is changing rapidly within SRP s service territory due to the program. External forces, such as the national ENERGY STAR program and the EISA legislation, are also impacting the CFL program. Cadmus recommends the program consider the current and near-term market in determining future plans. SRP FY10 Energy Efficiency Program: Evaluation Summary 25