Appendices Program Reviews and Benchmarking (LED, CFL, SUCH)
Appendices - Light Emitting Diode (LED) Program Program Goals and Design Process Goals are well defined and communicated, however, focus is on near-term (immediate) outcomes. Expanded vision, program logic assessment, and intentional identification and tracking of intermediate and longer-term market transformation indicators could prove useful along with more careful consideration of a prudent exit strategy. Budgets Well documented but based on conversion of 95% of all intersections to LED technology, does not seem to recognize the potential baseline level penetration of LEDs at existing intersections. Potential exists for lowering BC Hydro s contribution toward up front capital costs, elimination of plans to subsidize conversion of any new Ministry of Transportation (MOT) intersections (since they have mandate for LEDs), and reduction of 95% target to some lower (i.e., 60%), but defensibly sustainable level. Savings Estimates - BC Hydro energy savings estimates and measure lives associated with LED traffic light technologies are consistent with those used by other similar programs in North America. BC Hydro has not calculated demand savings for this program as other utilities do. Program Implementation Plans and Staffing Resources Well developed and communicated and sufficient for achieving current stated goals. Consider conducting process evaluation 6 months into program to help determine if program is being delivered as proposed, and to identify potential opportunities for efficiency and effectiveness improvements or program design modifications. Program Evaluation Plans Plans for tracking progress toward near-term (# of intersections) goal are thorough and database mechanisms seem to be in place. More effort could be focused on identifying and tracking intermediate and longer-term market transformation indicators. Cost Effectiveness Periodic benefit/cost analyses should be considered and expanded to recognize the significant and predictable capacity (demand reduction) benefits associated with LED traffic light technologies. Freeridership assumption of 3% should be re-assessed due to plans already in place (without the program) by the MOT in BC. Page A1
Appendices - Compact Fluorescent Light Bulb Program Program Goals and Design Process Program goals for the CCQ pilot are well defined. However, to ensure consistent understanding of key goals and objectives, a single page summary should be developed and made readily available for all staff and contractors. Quantity goals are typically well communicated but critical objectives such as education of retail staff and consumers can be lost in the effort to achieve the level of CFLs distributed. Savings Estimates Assumptions used in deriving the per CFL energy savings estimate of 69 kwh/year are consistent with those used in similar U.S. programs. Demand savings were mentioned in the evaluation plan and are based on wattage difference of standard bulb and CFL (initial estimate of 55 watt difference is consistent with programs reviewed). CFL measure life of 6.3 years is also consistent with other programs involving CFLs. Demand savings (KW) exist but have not been estimated, tracked, or reported. Budgets Total CCQ pilot budget appears reasonable but there was no documentation reflecting the breakdown of the budget. Assumed an average CFL cost of $12 yields an estimate of 38% of budget for rebates/discounts, which is reasonable for a pilot. Program Implementation Plans CCQ pilot plans are well developed and will offer an excellent template for Phase 2 of the CFL program. Program Evaluation Plans Plans for evaluating the CCQ pilot are detailed and comprehensive. Both process and impact areas seem to be well addressed. Consider adding a sales data reporting requirement to retailer Memorandum of Agreement (MOA) to assist in obtaining this critical but elusive information. Cost Effectiveness Benefit /cost analysis should be revisited in light of pilot evaluation results. Demand savings (KW) should be explicitly tracked and reported, and the value of these demand savings included in the cost effectiveness calculation for the TRC B/C ratio. Page A2
3c. Program Review - Schools, Universities, Colleges and Hospitals Program Goals and Design Process Goals are well defined and communicated, however, focus is on near-term (immediate) outcomes. A good set of program components are identified, however, the plans do not integrate the components in a strategic fashion. Expanded vision, program logic assessment, and intentional identification and tracking of intermediate and longerterm market transformation indicators could prove useful along with more careful consideration of a prudent exit strategy. More accurate and up to date market baseline and characterization research is needed. Budgets Well documented and indicates a reasonable split between the labor versus nonlabor shares of the budget. Staff resources committed to the project appear to be reasonable for the number of audits, projects, workshops, etc. indicated in the Implementation Plan. Savings Estimates In the first inspection by Audit Services, energy savings estimates for the retrofit component were found to be extremely optimistic. The original energy savings goals would have been difficult to achieve without specifically targeting highly energy intensive and larger facilities. In consultation with the Audit Services department, the basis for projected energy savings has been revised by Power Smart staff to be realistic but still aggressive. Demand savings (KW) at the overall program level have not been estimated, tracked, or reported. Program Implementation Plans and Staffing Resources Well developed, communicated, and sufficient for achieving current stated goals. Consider conducting process evaluation 6 months into program to help determine if program is being delivered as proposed, and to identify potential opportunities for efficiency and effectiveness improvements or program design modifications. Program Evaluation Plans Plans were not available for review. Cost Effectiveness Periodic benefit/cost analyses should be considered, and these analyses should include valuing demand savings. Particular attention should be paid to the Retrofit program. Refer to previous comments on Savings Estimates. Page A3
Appendices - Table 1 Table 1 Best Practices for Program Plans/Business Cases Program Plan Description of Program Plan LED CFL SUCH Understanding Market / Market Assessment Program plan provides a summary of the market, identifies the key market actors and decision makers, and provides citation to more detailed market assessment research Research done but not documented More research may be needed Understanding Market Actors Program plan describes the roles of key market actors in this market Need market characterization map Need market characterizatio n map and documentation of the actors Identified Market Barriers Program plan identifies the market barriers that hinder the adoption of energy efficiency measures and practices Assessing Efficient Technologies Program plan provides information on the efficiency measures and practices promoted by the program, measure energy and/or demand savings, useful lives, incremental capital and installation costs, etc. Page A4
Appendices Program Plan Program Logic presented & Indicators of market transformation identified Measurable and Verifiable Goals & Objectives Clearly Assigned Roles & Responsibilities for Implementation/Staffing Plan Operational Procedures Plan Description of Program Plan LED CFL SUCH A logic model for a program includes each of the following elements: the structure and decision influences currently operating in the market the intervention design and its underlying logic the expected results of the intervention (program) including a description of what will happen and how the program will achieve the desired outcomes (including indicators of market transformation, mileposts, penetration expected with and without the program, etc.) an explicit description of what should happen under different contingency situations (program ramp-up, ramp-down, etc.), and a description of what indictors can be used to signal the need for changes in program strategy The program plan clearly states the goals and objectives of the program, and these goals and objectives are reasonable, understandable, measurable, believable and achievable (RUMBA) The program plan explains what key strategies will be deployed, who will implement these strategies, and will explain the staffing resources needed for the program The program plan describes necessary operational procedures (for example, the RFP development and issue process, the incentive application and approval process, procedures for telemarketing or direct mail, approval process for the program budget, procedures for data collection and tracking, etc.) Need mid-term and longer term indicators of market transformation Need tracking of longer-term indicators of market transformation Goals need to be stated in one place too scattered Goals are realistic but aggressive Page A5
Appendices Program Plan Plan for Addressing Training Requirements Marketing Strategy Assessment of Risks / Contingency Planning Well-Defined Short and Long Term Budget Assessment of Benefit/Cost Analysis and Projected Savings Transition Strategy Operational Procedures Plan Description of Program Plan LED CFL SUCH The program plan describes training requirements for program staff, trade allies, program participants, and other key market actors The plan describes all marketing strategies to be used (direct mail, telemarketing, public relations, trade shows, one on one site visits, incentives, etc.) The program plan describes key risks to the program and contingency plans for key risks The program plan provides the annual budget for the program for one to three years The program plan presents a cost effectiveness analysis for the program according to BC Hydro required cost effectiveness tests (the Total Resource Test >= 1; utility cost per kwh saved < $.02 (Canadian $) A transition strategy explains how a program will evolve, ramp up or ramp down based upon movements in indicators of market transformation over time The program plan describes necessary operational procedures (for example, the RFP development and issue process, the incentive application and approval process, procedures for telemarketing or direct mail, approval process for the program budget, procedures for data collection and tracking, etc.) Follow through required Demand savings need to be recognized and valued. Freeridership assumption of 3% needs to be re-assessed (too low) Need plan to lock in savings (plan for sustainability) No documentation available to review Demand savings need to be recognized and valued. Need plan to lock in savings (plan for sustainability) Page A6
Appendices - Table 2 TABLE 2 Best Practices for Program Implementation Plans Program Implementation Encourage Sustainable Market Clear Program Schedule Description LED CFL SUCH Implementation strategies are targeted to create a market that will sustain energy efficiency awareness and energy efficient purchasing decisions and practices A detailed program schedule with key milestones exists Needs more attention Needs more attention Budget Tracking Mechanism in Place Operational Procedures in Place/Flow charts Competitive Bidding Procedures are in place to track and report to BC Hydro management actual program expenditures versus budgeted amounts on a monthly basis Clear operating procedures exist for major processes, major processes have been flow-charted, staff has received training on all operating procedures (the RFP development and issue process, the incentive application and approval process, procedures for telemarketing or direct mail, approval process for the program budget, procedures for data collection and tracking, etc.). Data tracking is in place to monitor cycle times of key processes and to ensure that backlogs are minimized or eliminated. Attention given to continuous improvement of processes. Contracts for outside services are accomplished through a well-defined competitive bidding process (RFP process has been flowcharted, operating procedures exist, staff has been trained on these procedures) Need to monitor cycle times for key processes to eliminate backlogs and bottlenecks Need to monitor cycle times for key processes to eliminate backlogs and bottlenecks Need to monitor cycle times for key processes to eliminate backlogs and bottlenecks Page A7
Appendices Program Implementation M&V Plan / Output Tracking / Collection of Necessary Program Data Feedback Loop to Facilitate Interim Design Modifications based on Field Experience Description LED CFL SUCH Data tracking mechanisms are in place to track program activity and outputs (number of participants, trade ally partners, identified energy savings, committed savings, implemented savings, implemented and verified savings) The program and/or implementation plan explains how program evaluation results will be reviewed and used to make mid-course corrections in program design, targeting, strategies, etc. Needs more work; M&V plans will be written Need to monitor mid and longerterm indicators of MT Needs more work; M&V plans will be written Need to monitor mid and longerterm indicators of MT Needs more work; M&V plans will be written Need to monitor mid and longerterm indicators of MT Page A8
Appendices - Table 3 TABLE 3 Best Practices for Program Evaluation Plans Program Evaluation Plan Program evaluation goals Identify Key Indicators of MT Compile Baseline Data on Indicators Develop Market Progress Reports Track and Assess Market Transformation Indicators Description LED CFL SUCH The evaluation plan for a program presents the overall goals of the evaluation (process analysis, impact analysis, market indicator analysis, benefit/cost analysis, etc.), focusing on the audience for the report The evaluation plan describes the key indicators of market transformation that are being tracked for a program The plan describes how the baseline for indicators of market transformation or for energy savings measurement will be accomplished Market progress reports summarize periodic or on-going market assessment and measurement to provide feedback for the logic model, refining program interventions as necessary and documenting progress to date. Progress reports focus on changes that have occurred in a market (not program outputs) and attribution of those changes. Once indicators of market transformation have been identified, the evaluation plan describes the necessary research approaches and methods to collect indicator data over time. The research approaches to be used may include consumer surveys, depth interviews, content analyses and secondary research. Evaluation plan needs to be developed Evaluation plan needs to be developed Page A9
Appendices Program Evaluation Plan Process evaluation plan Impact evaluation plan Risk Assessment Periodic Review of Cost Effectiveness Description LED CFL SUCH Process evaluations examine the efficiency and effectiveness of program delivery, including the effectives of communication among program staff, program participants, non-participants and key market actors. Process evaluations ensure that program delivery is continuously improved, assess whether customers are satisfied with products and services provided, assess any program backlogs or bottlenecks, and whether program communications are effective. Impact evaluations assess the energy benefits, non-energy benefits, and market effects due to a program, and are thus essential for determining the cost effectiveness of a program. An assessment to the major risks and uncertainties that may affect the successful implementation of a program. These risks include internal and external events, controllable and non-controllable events. A detailed cost effectiveness analysis examines whether program benefits on a life cycle basis are likely to exceed program costs Page A10