Natural Gas Demand Side Management Evaluation, Measurement, and Verification (EM&V) Plan

Transcription

1 Natural Gas Demand Side Management Evaluation, Measurement, and Verification (EM&V) Plan submitted to the Ontario Energy Board Date: November 10, 2016

2 DNV GL - Energy

3 Table of contents EXECUTIVE SUMMARY... V ES.1 Scope v ES.2 Methodology vi ES.3 Summary of plan vii 1 INTRODUCTION Scope Methodology Document organization 4 2 PORTFOLIO OVERVIEW Overview of portfolio Summary of portfolio by regulatory program category Resource acquisition programs Large Volume program Low income programs Market transformation and performance-based programs , 2016, and 2017 portfolio by program Portfolio summary PROGRAM EVALUATION PLANS Commercial and Industrial Custom, Large Volume, Run it Right, and Low Income Multi-family programs Commercial and Industrial Prescriptive program Residential Savings by Design, Optimum Home, and Commercial Savings by Design programs AND 2017 PROGRAM EVALUATION PLANS Home Energy Conservation () Home Reno Rebate (Union) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Billing analysis Participant survey and trade ally interviews Market studies Other options Residential Adaptive Thermostats () Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Billing analysis Participant survey Market studies 42 DNV GL 11/10/16 Page i

4 4.3 Home Winterproofing () Home Weatherization, Furnace End-of-Life, and Aboriginal Offerings (Union) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Billing analysis Participant survey and trade ally interviews Market studies Other options Low Income Multi Residential Affordable Housing () Low-Income Multi- Family Offering (Union) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Participant survey and trade ally interviews Market studies Other options Low Income New Construction () Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Desk review of files On-site inspections Participant survey and trade ally interviews Market studies Large Volume (Union) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Savings verification Participant surveys Market assessment Persistence study Knowledge and understanding of other programs projects Commercial and Industrial Prescriptive Offering () Prescriptive and Quasi- Prescriptive Offering (Union) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Tracking data review and certification Savings verification Participant surveys Market assessment Persistence study Knowledge and understanding of other programs projects Commercial and Industrial Direct Install Programs ( and Union) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation 80 DNV GL 11/10/16 Page ii

5 Tracking data review and certification Savings verification Participant surveys Persistence study Commercial and Industrial Custom Program ( & Union) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Savings verification Participant surveys Market assessment Persistence study Knowledge and understanding of other programs projects Small Commercial New Construction Program ( Pilot) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Residential Savings by Design () Optimum Home (Union) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Verification of program metrics Market effects analysis Commercial Savings by Design Programs ( and Union) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Verification of program metrics Market effects: baseline study Run it Right () RunSmart (Union) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Verification of program metrics Include Run it Right / Run Smart effects in NTG studies Comprehensive Energy Management () Strategic Energy Management (Union) Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation Include CEM/SEM effects in NTG studies Verification of program metrics Market effects: baseline study School Energy Competition () Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation 107 DNV GL 11/10/16 Page iii

6 Verification of program metrics Energy Literacy Program () Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation General population survey Energy Leaders Pilot Program () Program description Key objectives Program risks, impacts, and targeted evaluation activities Evaluation activity recommendation NTG comparison CROSS-CUTTING CONSIDERATIONS Market assessment studies Coordination with other research and evaluation activities Process evaluation Persistence studies 114 APPENDIX A. APPENDIX B. APPENDIX C. APPENDIX D. APPENDIX E. APPENDIX F. LIST OF FIGURES AND TABLES EVALUATION MEMOS HISTORIC SAVINGS AND BUDGET LOGIC MODELS CUSTOMER INCENTIVES AND OTHER SERVICES PROCESS FLOW DNV GL 11/10/16 Page iv

7 EXECUTIVE SUMMARY This document has been prepared for the Ontario Energy Board (OEB) and outlines the Evaluation, Measurement & Verification (EM&V) plan related to Gas Distribution Inc. s () and Union Gas Limited s (Union) natural gas demand-side management (DSM) programs delivered in 2015, 2016, and The outcome of the exercise is a list of prioritized evaluation activities to be completed in 2016, 2017, and The overall objectives of the evaluations are to: Assess portfolio impacts for the purpose of determining annual savings results, shareholder incentive and lost revenue amounts, and future year targets. Assess the effectiveness of energy efficiency programs on their participants and/or market, including results on various scorecard items. Identify ways in which programs can be changed or refined to improve their performance. To accomplish those objectives, the Evaluation Contractor (EC) team has identified the components of each program that introduce the greatest uncertainty (risk) to the program s natural gas savings estimate or expected shareholder incentive. We then identified evaluation activities to address those risks and prioritized the activities into each of the three evaluation years. ES.1 Scope This evaluation plan addresses the DSM efficiency programs approved by the OEB for and Union for the 2015 to 2020 period. The plan addresses programs offered in 2015, 2016, and The evaluations will be started in 2016 and continue through 2017 and The evaluation types in the plan include: The verification of scorecard metrics Net-to-gross evaluations, including free ridership and spillover Market assessment and market transformation studies Options to address process evaluation and cost effectiveness Table ES - 1 shows the programs that have been approved for each utility for 2015, 2016, and 2017; 2015 was a transition year, bridging the former DSM term ( ) with the new DSM Framework ( ). Therefore, there are differences between 2015 and programs, and the associated annual evaluation activities taking place in 2016 and Table ES - 1. Programs approved for 2015, 2016, and 2017 Program Residential Home Energy Conservation / Home Reno Rebate Union Union Union Energy Savings Kit Union Residential Adaptive Thermostats C&I Custom Union Union Union DNV GL 11/10/16 Page v

8 Program C&I Prescriptive Union Union Union C&I Direct Install Union Union C&I Small Commercial New Construction Large Volume Union Union Union LI Home Winterproofing/ Weatherization Union Union Union LI Multi-Res Affordable Housing Union Union Union LI New Construction LI Aboriginal Union LI Furnace End of Life Union Union Residential Savings by Design / Optimum Home Union Union Union Run it Right / Run Smart Union Union Commercial Savings by Design Union Union Strategic / Comprehensive Energy Management Union Union School Energy Competition Energy Leaders Pilot Energy Literacy Program Home Labelling ES.2 Methodology To identify the evaluation activities, the EC used a value of information framework, which followed the steps outlined in Table ES - 2. The process allowed the EC to identify uncertainty (risk) in the natural gas savings estimate and scorecard metrics and prioritize evaluation activities to mitigate the risks to the greatest degree. DNV GL 11/10/16 Page vi

9 Table ES - 2. Steps in completing the value of information exercise Process Step Purpose Notes 1. Characterize the portfolio 2. Characterize the programs 3. Identify and classify risks 4. Produce evaluation activities and relative cost 5. Prioritize evaluation activities for 2016 through 2018 Provide comparison metrics for each program. Define the programs and the elements that will reveal risks. Identify the risks (or uncertainties) to the accuracy of the savings and shareholder incentive. Identify the evaluation activities that address the risks. Identify which evaluation activities should be completed and when. The selection of comparison metrics allow the EC to determine the importance of one program relative to another program. The characterization includes target market, past evaluation activities, maturity, historic budget and savings, key objectives, scorecard metrics, process flow, and program offerings. In total, this provides the information necessary to identify threats (or risks) to savings and shareholder incentive accuracy and a general sense of the cost of the activity necessary to address that risk. The identification produces a comprehensive list of the risks that may affect the accuracy of the program s savings estimate and shareholder incentive. The risks are classified by the likelihood they will happen and the potential impact they would have on the savings or incentive. These are the evaluation activities that, if completed, should mitigate the risks identified in the previous step. Their cost is identified at a very general (high, medium, low) level. This step prioritizes activities (based on highest risk, highest impact, and EC judgment within programs and the impact of the program within the portfolio) and groups them (where logical) into studies to be completed in 2016, 2017, or ES.3 Summary of plan Table ES - 3 and Table ES - 4 show a list of the Evaluation Contractor (EC) recommended evaluation activities by year through the mid-term review. Table ES - 3 shows the evaluation activities that address specific programs with the evaluation priority level for that program, a list of the recommended evaluation activities, and the timing for each activity in 2016, 2017, or 2018 calendar years. The year associated with each evaluation is the year in which the EC proposes that the study begin. It does not relate to the program year being evaluated, or the year in which the study will be completed. Such information (program year evaluated and completion date) will depend on the type DNV GL 11/10/16 Page vii

10 of program being evaluated and the start date of the evaluation. For example, a typical residential billing analysis relies on at least one year of billing data after the measure has been installed; therefore, a residential billing analysis initiated in 2017 likely will not address 2016 program participants. However, participant surveys are best completed as soon as possible after a measure is installed (depending on what is being measured), so a free ridership survey initiated in 2017 likely would address 2016 and possibly 2017 program participants. Additional detail on the recommended evaluation activities can be found in each program section. Table ES - 4 shows additional evaluation activities that cut across multiple programs. DNV GL 11/10/16 Page viii

11 Table ES - 3. Summary of evaluation plan by program and year Program(s) Evaluation Priority Level Recommended Evaluation Activities Residential Home Energy Conservation / Home Reno Rebate LI Home Winterproofing/ Weatherization LI Furnace End of Life High 40% of budget ~10% of savings 33% of shareholder incentive Billing analysis (BA) to estimate energy savings Desk reviews to compare BA to program-estimated savings through building simulation Net-to-gross (NTG) evaluation using participant surveys to verify occupancy assumptions in simulation model Market assessment of preretrofit building characteristics, base load assumptions, furnace baseline assumptions, incremental cost, measure life. Billing analysis to estimate energy savings Desk reviews of online thermostat data NTG evaluation using participant surveys to verify pre-installation thermostat type and household behavior Market assessment of incremental cost and measure life. Billing analysis Desk Reviews NTG evaluation Market assessment Residential Adaptive Thermostats Medium Measureable savings and shareholder incentive New program Billing analysis Desk Reviews NTG evaluation Market assessment DNV GL 11/10/16 Page ix

12 Program(s) Evaluation Priority Level Recommended Evaluation Activities C&I Custom Large Volume High 75% of savings 35% of shareholder incentive Savings verification to confirm measure installation and verify the energy savings assumptions NTG evaluation using participant surveys which also address incremental cost Savings verification NTG evaluation Savings verification NTG evaluation Savings verification NTG evaluation C&I Prescriptive High Approximately 10% of savings and shareholder incentive Installation verification to confirm measure installation Tracking certification to ensure proper application of TRM savings NTG evaluation using participant surveys which also address incremental cost Tracking certification Installation verification Tracking certification NTG evaluation Installation verification Tracking certification C&I Direct Install Medium Measureable savings and shareholder incentive Installation verification with on-sites that confirm continued installation Tracking certification to ensure proper application of TRM savings NTG evaluation using participant surveys to verify operating characteristics for savings estimates Installation verification Tracking certification NTG evaluation Tracking certification C&I Small Commercial New Construction Low Small savings Small shareholder incentive New program Savings verification with telephone surveys and selective on-sites to confirm energy savings NTG evaluation using participant surveys Savings verification NTG evaluation DNV GL 11/10/16 Page x

13 Program(s) Evaluation Priority Level Recommended Evaluation Activities LI Multi-Res Affordable Housing High 4 th largest in shareholder incentive Savings verification with on-sites to confirm energy savings in 2016 Reassessment of verification process in 2017 to consider telephone review only Market assessment of preretrofit building characteristics, base load assumptions, heating baseline assumptions, incremental cost, measure life. Savings verification Reassessment of savings verification Market assessment LI New Construction Low Small savings Small shareholder incentive New program Savings verification with desk reviews to confirm energy savings Market assessment of the baseline for new housing units, incremental cost, measure life. Savings verification Market assessment Residential Savings by Design / Optimum Home Commercial Savings by Design Medium Measureable shareholder incentive Metrics verification using telephone surveys to confirm participation Market assessment to determine builder and owner baseline energy efficiency awareness and penetration Metrics verification Market assessment Run it Right / Run Smart Medium Measureable shareholder incentive Overlap with C&I Custom and Prescriptive Metrics verification using telephone surveys to confirm participation NTG evaluation using participant surveys which also address incremental cost (Run it Right only) Metrics verification NTG evaluation (RiR only) Metrics verification DNV GL 11/10/16 Page xi

14 Program(s) Evaluation Priority Level Recommended Evaluation Activities Strategic / Comprehensive Energy Management School Energy Competition Energy Leaders Pilot Energy Literacy Program LI Aboriginal Energy Savings Kit Home Labelling Medium Measureable shareholder incentive Overlap with C&I Custom and Prescriptive Low No savings Small shareholder incentive Low No or small savings No or small shareholder incentive Some programs no longer offered Some programs new At minimum, the EC will annually confirm the scorecard metrics for each program. Metrics verification using telephone surveys to confirm participation Market assessment to determine C&I customer energy efficiency awareness and penetration Metrics verification using telephone surveys to confirm participation No evaluation activities are recommended for these programs. Metrics verification Metrics verification Market assessment Metrics verification None None None DNV GL 11/10/16 Page xii

15 Table ES - 4. Summary of cross-cutting evaluation activities by year Program(s) Study Manager Evaluation Priority Level Recommended Evaluation Activities C&I Boiler Baseline Study Utilities High Currently in progress Market assessment Residential Persistence Study OEB Medium Create a roadmap that identifies priorities Market assessment Market assessment Commercial and Industrial Persistence Study OEB High Create a roadmap that identifies priorities Market assessment Market assessment DNV GL 11/10/16 Page xiii

16 1 INTRODUCTION This document has been prepared for the Ontario Energy Board (OEB) and outlines the Evaluation, Measurement & Verification (EM&V) plan related to Gas Distribution Inc. s () and Union Gas Limited s (Union) natural gas demand-side management (DSM) programs delivered in 2015, 2016, and The outcome of the exercise is a list of prioritized evaluation activities to be completed in 2016, 2017, and The overall objectives of the evaluations are to: Assess portfolio impacts for the purpose of determining annual savings results, shareholder incentive and lost revenue amounts, and future year targets. Assess the effectiveness of energy efficiency programs on their participants and/or market, including results on various scorecard items. Identify ways in which programs can be changed or refined to improve their performance. To accomplish those objectives, the EC team has identified the components of each program that introduce the greatest uncertainty (risk) to the program s natural gas savings estimate or expected shareholder incentive. We then identified evaluation activities to address those risks and prioritized the activities into each of the three evaluation years. The remaining portions of section 1 provide an overview of the methodology used to identify the evaluation activities and the organization of the remainder of the document. 1.1 Scope This evaluation plan addresses the natural gas DSM programs approved by the OEB for and Union for the 2015 to 2020 program years. 1 The evaluation plan is for programs offered in 2015, 2016, and The evaluations will be started in 2016, 2017, and The plan addresses studies that will be completed in addition to the already-contracted EC team responsibilities, which include oversight of all evaluations, an annual review and update(s) of the technical reference manual (TRM), an annual report summarizing all evaluation activities and results, and a cost effectiveness assessment of the utility programs. The evaluation types in the plan include: The verification of scorecard metrics Net-to-gross evaluations, including free ridership and spillover Market assessment and market transformation studies Options to address process evaluation and cost effectiveness Table 1-1 shows the programs that have been approved for each utility for 2015, 2016, and 2017; 2015 was a transition year, bridging the former DSM term ( ) with the new DSM Framework ( ). Therefore, there are differences between 2015 and evaluations. 1 OEB Decision and Orders, EB /EB dated January 20, 2016 and February 24, DNV GL 11/10/16 Page 1

17 Table 1-1. Programs approved for 2015, 2016, and 2017 Program Residential Home Energy Conservation / Home Reno Rebate Union Union Union Energy Savings Kit Union Residential Adaptive Thermostats C&I Custom Union Union Union C&I Prescriptive Union Union Union C&I Direct Install Union Union C&I Small Commercial New Construction Large Volume Union Union Union LI Home Winterproofing/ Weatherization Union Union Union LI Multi-Res Affordable Housing Union Union Union LI New Construction LI Aboriginal Union LI Furnace End of Life Union Union Residential Savings by Design / Optimum Home Union Union Union Run it Right / Run Smart Union Union Commercial Savings by Design Union Union Strategic / Comprehensive Energy Management Union Union School Energy Competition Energy Leaders Pilot Energy Literacy Program Home Labelling 1.2 Methodology To identify the evaluation activities, the EC used a value of information framework, which followed the steps outlined in Table 1-2. The process allowed the EC to identify uncertainty (risk) in the natural DNV GL 11/10/16 Page 2

18 gas savings estimate and scorecard metrics and prioritize evaluation activities to mitigate the risks to the greatest degree. 1. Portfolio characterization. The process started at the broadest level (the portfolio, the Union portfolio, and the two portfolios combined) to identify program priorities based on comparison metrics. The portfolio characterization allowed the EC to determine the programs that had the greatest impact on the portfolio-level natural gas savings estimate and shareholder incentive. 2. Program characterization. The next step was to characterize each program individually in terms of target market, past evaluation activities, program maturity, historic budget and savings, key objectives, scorecard metrics, process flow, and program offerings. This characterization provided the information necessary to identify the components that may cause uncertainty (risk) in the program s natural gas savings estimate and shareholder incentive. 3. Identify program risks. The next step was to identify the specific uncertainties (risks) and classify them according to their likelihood of occurring and the level of impact they might have on the program-level savings estimate and shareholder incentive. For example, if the energy savings estimate for an unpopular measure is known to use outdated information, then the risk is likely to occur but the impact is low, because the measure is not installed very often. 4. Produce evaluation activities. The EC then identified evaluation activities to address each risk, and the relative cost associated with the activities. The combination of likelihood, risk, and cost allowed us to combine evaluation activities into studies and prioritize them into individual evaluation years by program. 5. Final prioritization. Returning to the portfolio characterization, the EC was able to use the comparison metrics to identify the greatest risks to the portfolio as a whole, to prioritize evaluation activities across programs. The final result is an evaluation plan that identifies activities to be completed in 2016, 2017, and Once the steps in Table 1-2 were completed, the EC sought comments from the OEB and Evaluation Advisory Committee (EAC) on the planned evaluation activities and timeline. Table 1-2. Steps in completing the value of information exercise Process Step Purpose Notes 1. Characterize the portfolio 2. Characterize the programs Provide comparison metrics for each program. Define the programs and the elements that will reveal risks. Comparison metrics allow the EC to determine the importance of one program relative to another program. The characterization includes target market, past evaluation activities, maturity, historic budget and savings, key objectives, scorecard metrics, process flow, and program offerings. In total, this provides the information necessary to identify threats (or risks) to savings and shareholder incentive accuracy and a general sense of the cost of the activity necessary to address that risk. DNV GL 11/10/16 Page 3

19 Process Step Purpose Notes 3. Identify and classify risks 4. Produce evaluation activities and relative cost 5. Prioritize evaluation activities for 2016 through 2018 Identify the risks (or uncertainties) to the accuracy of the savings and shareholder incentive. Identify the evaluation activities that address the risks. Identify which evaluation activities should be completed and when. The identification produces a comprehensive list of the risks that may affect the accuracy of the program s savings estimate and shareholder incentive. The risks are classified by the likelihood they will happen and the potential impact they would have on the savings or incentive. These are the evaluation activities that, if completed, should mitigate the risks identified in the previous step. Their cost is identified at a very general (high, medium, low) level. This step prioritizes activities (based on highest risk, highest impact, and EC judgment within programs and the impact of the program within the portfolio) and groups them (where logical) into studies to be completed in 2016, 2017, or Document organization The remainder of this document is organized with a section to classify the portfolio, a section addressing 2015 evaluation activities, a section addressing 2016 and 2017 evaluation activities, and a section which discusses evaluation activity areas that span multiple programs (or cross-cutting considerations). The portfolio section (section 2) summarizes the Union and portfolios by program within four regulatory categories: 2 resource acquisition programs, large volume programs, low income programs, and market transformation programs. The portfolio section of the report provides comparison metrics to help prioritize each category and program within the overall portfolio. Section 4 includes one section for each program (or substantially similar programs offered by both Union and ). Within the program sections, there are sub-sections that discuss the following areas: Program sub-section 1: describes the program Program sub-section 2: identifies the key objectives of the program Program sub-section 3: describes in a table the risks associated with the program, the likelihood of that risk occurring, the potential impact of the risk, the evaluation activity that can mitigate the risk, and the relative evaluation cost associated with that activity 2 These program areas generally share the same characteristics, with the central goal of reducing natural gas consumption, but have been separated to align with the OEB s DSM Framework policy document and the subsequent OEB Decisions and Orders related to and Union s DSM Plans. DNV GL 11/10/16 Page 4

20 Program sub-section 4: provides a recommended evaluation activity with a high-level overview of the options and the tasks necessary to complete the activities. The cross-cutting section includes a discussion of: Market assessment / market effects research Coordination with evaluation activities being undertaken by other entities, including Union,, and the IESO Measure persistence studies DNV GL 11/10/16 Page 5

21 2 PORTFOLIO OVERVIEW Section 2 provides an overview of the 2015, 2016, and 2017 portfolios for, Union and the two utilities combined. The objective of this section is to provide comparison metrics that allow the EC team to weigh the relative importance of one program against another within the portfolio, judged by the impact the program has on the portfolio-level savings estimate and shareholder incentive. The comparison metrics we have identified are: Planned cumulative natural gas savings. Cumulative (or lifetime) natural gas savings must be accurate to ensure the utility s performance is accurate assessed against OEB-approved targets. Further, accurate savings aid load forecasting, ensure accurate lost revenue and shareholder incentive calculations, and facilitate accurate cost effectiveness calculations. An accurate measure of cost effectiveness is important to ensure that ratepayer money is being used efficiently. Planned budget. Budget is an input to cost-effectiveness and will help identify whether utility funds are spent wisely. Maximum available shareholder incentive. Most programs have a scorecard with metrics that, if met, will result in financial incentives paid to the utilities. The maximum available shareholder incentive is the amount they will receive if 150% of the scorecard metrics are met. Although both budget and shareholder incentive are included as a comparison metric, it is important to note that they are not independent. The OEB directed 3 the utilities to allocate the shareholder incentive to programs that will achieve significant gas savings, based on the budget amounts for each program, and using scorecard metrics that will encourage and achieve key priorities. Section 2 is designed to characterize the portfolio in increasing levels of detail. The subsections will describe the following information: Section 2.1 provides an overview of the, Union, and combined portfolios in terms of cumulative savings, budget, and shareholder incentive. Section 2.2 shows the distribution of savings, budget, and shareholder incentive across the four regulatory program categories of resource acquisition, large volume, low income, and market transformation. Section 2.3 shows the distribution of savings, budget, and shareholder incentive across all programs in the resource acquisition category. Section 2.4 shows the distribution of savings, budget, and shareholder incentive for the large volume category. Section 2.4 shows the distribution of savings, budget, and shareholder incentive across all programs in the low income category. Section 2.6 shows the distribution of savings, budget, and shareholder incentive across all programs in the market transformation category. Section 2.7 has tables that summarize the information, by individual program, and across all programs for 2015, 2016, and It also has a table that indicates how long the program has 3 Ontario Energy Board. Filing Guidelines to the Demand Side Management Framework for Natural Gas Distributors ( ). December 22, Section 5.2. DNV GL 11/10/16 Page 6

22 been offered, whether there is a similar program in the Ontario electricity conservation framework, and the recent evaluations conducted on each program. Section 2.8 summarizes the data presented in section Overview of portfolio Table 2-1 shows the magnitude of the claimed (2015) or planned (2016 and 2017) cumulative savings, claimed (2015) or planned (2016 and 2017) budget, and claimed (2015) or maximum available (2016 and 2017) shareholder incentive by program year for, Union, and the two utilities combined. The maximum available shareholder incentive is the amount the utilities will receive if they achieve 150% of their scorecard metrics. The planned savings are based in part on a formula that assumes that the utilities achieve their energy savings targets from the previous years. The available shareholder incentive is the same for each utility and fixed for every program year in the current ( ) planning cycle. In 2015, on a pre-verification basis, Union has claimed $7.5 million in shareholder incentive while has claimed $10.3 million. The Union savings goals (including savings from its Large Volume program) are approximately 2.5 times the savings goals, although the Union budget is slightly smaller. The nature of the utilities customer profile and program offerings lead to differences in savings goals and budget amounts. The planned cumulative savings (cumulative cubic meters of natural gas, or CCM) grows by more than 35% from 2015 to 2016 and approximately 10% from 2016 to The budget grows by more than 60% from 2015 to 2016 and approximately 15% from 2016 to DNV GL 11/10/16 Page 7

23 Table 2-1. Planned or claimed cumulative savings and budget and maximum available or claimed shareholder incentive Utility Cumulative Net Natural Gas Savings, CCM Program Budget* Maximum Shareholder Incentive (claimed (2015) or if 150% of target achieved) 2015 Claimed Union 1,568,082,914 $22,756,466 $7,548, ,616,755 $27,350,814 $10,318,594 Total 2,433,699,669 $50,107,280 $17,866, Planned and Maximum Available Union 2,237,676,855 $36,839,000 $10,450,000 1,080,480,685 $45,120,096 $10,450,000 Total 3,318,157,540 $81,959,096 $20,900, Planned and Maximum Available Union 2,202,766,209 $43,232,000 $10,450,000 1,189,857,633 $51,641,883 $10,450,000 Total 3,392,623,842 $94,873,883 $20,900,000 * Note: Program budget totals do not include overheads. Source: Ontario Energy Board. (2016). Decision and Order EB /EB , Chapter 5. Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Summary of portfolio by regulatory program category Table 2-2 shows the distribution of the claimed (2015) or planned (2016 and 2017) cumulative savings, actual (2015) or planned (2016 and 2017) budget, and claimed (2015) or total available (2016 and 2017) shareholder incentive across the regulatory program categories of resource acquisition, large volume, low income, and market transformation for 2015, 2016, and By definition, market transformation programs do not include energy savings metrics. 4 4 s Run it Right and Comprehensive Energy Management programs have targets on both the market transformation and resource acquisition scorecards with small energy savings goals. DNV GL 11/10/16 Page 8

24 Table 2-2. Distribution of planned or claimed cumulative savings and budget and total available or claimed shareholder incentive across program categories Utility % of Planned or Claimed Cumulative Savings % of Planned or Claimed Budget % of Maximum Shareholder Incentive (claimed (2015) or if 150% of target achieved) 2015 RA 1 LV 2 LI 3 MT 4 Total RA 1 LV 2 LI 3 MT 4 Total RA 1 LV 2 LI 3 MT 4 Total Union 60% 37% 3% 0% 100% 57% 10% 29% 4% 100% 63% 0% 29% 8% 100% 88% 0% 11% < 1% 100% 61% 0% 24% 15% 100% 63% 0% 17% 21% 100% Combined 70% 24% 6% < 1% 100% 59% 4% 26% 10% 100% 63% 0% 22% 15% 100% 2016 Union 52% 45% 2% < 1% 100% 61% 9% 26% 4% 100% 61% 9% 25% 5% 100% 91% 0% 9% < 1% 100% 61% 0% 23% 16% 100% 65% 0% 23% 12% 100% Combined 64% 31% 4% < 1% 100% 61% 4% 24% 11% 100% 63% 5% 24% 9% 100% 2017 Union 58% 39% 3% < 1% 100% 62% 7% 25% 6% 100% 61% 9% 25% 5% 100% 91% 0% 9% < 1% 100% 63% 0% 21% 16% 100% 65% 0% 23% 12% 100% Combined 69% 26% 5% < 1% 100% 63% 3% 23% 11% 100% 63% 5% 24% 9% 100% Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. 1 RA = resource acquisition 2 LV = large volume 3 LI = low income 4 MT = market transformation DNV GL 11/10/16 Page 9

25 There is a large change in the proportion of energy savings between 2015 and 2016 while the distribution of savings between 2016 and 2017 is approximately the same. This is to be expected as 2015 acted as a transition year between the and frameworks with budgets and priorities similar to those in Due to differences in the rate class structure of and Union, Union delivers a large volume DSM program. Because of the large volume program category, the savings distribution for Union and are very different. In 2016 and 2017, almost half of Union s targeted energy savings come from the large volume category. The other significant savings proportion goes to resource acquisition, with 52% of savings in 2016 and 58% of savings in has a slightly larger portion of savings going to low income at 9%. The change in budget allocation between 2015 and 2016 is not as drastic as the change in savings distribution. In 2015, the portion of the budget spent on resource acquisition programs (combined) was 59%; in 2016 the planned budget is 61% of the portfolio and in 2017, 63%. Across utilities, the budget allocation is largely the same for resource acquisition and low income, but Union assigns 9% of its 2016 budget to large volume, while assigns a larger portion of its budget to market transformation. As stated previously, the shareholder incentive distribution across regulatory categories is approximately the same as the budget distribution. Figure 2-1 shows the distribution of claimed budget, savings, and shareholder incentive for 2015 and the distribution of planned budget, savings, and shareholder incentive for 2016 for both utilities combined. Since the distribution across 2016 and 2017 is approximately equal for both utilities, Figure 2-1 can be considered representative of The table and figures show that the majority of the budget, savings, and shareholder incentive are targeted toward the resource acquisition programs, consistent with the OEB s DSM Framework. Low income programs account for a large portion of budget and shareholder incentives, while the large volume program accounts for a large portion of savings. Market transformation programs are on the lower end of all three categories. These data suggest that evaluation resources should be first targeted toward resource acquisition programs, then low income and large volume, and finally market transformation. The exception might be for market transformation programs that are expected to produce significant spillover savings. DNV GL 11/10/16 Page 10

26 Figure 2-1. Distribution of combined planned or claimed cumulative savings and budget, savings, and total available or claimed shareholder incentive across program categories Source: Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Ontario Energy Board. (2016). Decision and Order EB /EB Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. DNV GL 11/10/16 Page 11

27 2.3 Resource acquisition programs Figure 2-2, Figure 2-3, and Figure 2-4 show the percentage of the claimed (2015) or planned (2016 and 2017) portfolio savings, claimed (2015) or planned (2016 and 2017) portfolio budget, and claimed (2015) or total available (2016 and 2017) portfolio shareholder incentive for the 2015, 2016, and 2017, Union, and combined resource acquisition programs. Overall, the commercial and industrial (C&I) programs are much larger than the residential programs. s C&I programs (Custom, Prescriptive, Direct Install, and Small Commercial New Construction) and residential programs (Home Energy Conservation and Adaptive Thermostats) each receive approximately the same portion of portfolio budget at roughly 30%. The majority of savings (approximately 75%) come from the C&I programs. The OEB approved three new programs to begin implementation in 2016: Adaptive Thermostats, Small Commercial New Construction, and Direct Install. Figure resource acquisition programs C&I Small Commercial New Construction C&I Direct Install C&I Prescriptive C&I Custom Adaptive Thermostats 2017 % Total SI 2017 % Total Savings 2017 % Total Budget 2016 % Total SI 2016 % Total Savings 2016 % Total Budget 2015 % Total SI 2015 % Total Savings 2015 % Total Budget Res Home Energy Conservation 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. Where necessary, shareholder incentive and budget are distributed within programs based on the proportion of energy savings for that program. Similar to, Union s C&I programs (Custom, Prescriptive, Direct Install) account for a large portion of its total portfolio budget (approximately 40%) and savings (more than 50%). As ordered by the OEB, Union will no longer deliver the Energy Savings Kit program beyond As a portion of the overall budget, the Home Reno Rebate program more than doubled from 9% in 2015 to 20% in The Direct Install program was added in 2016, and its budget is expected to grow drastically in 2017 (from 1% to 6%), but the savings remain small (1-2%). DNV GL 11/10/16 Page 12

28 Figure Union resource acquisition programs Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. Where necessary, shareholder incentive and budget are distributed within programs based on the proportion of energy savings for that program. Figure Combined resource acquisition programs Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. Where necessary, shareholder incentive and budget are distributed within programs based on the proportion of energy savings for that program. 2.4 Large Volume program As a regulatory category, the Large Volume category is unique in that it is made up of only one program offered by one utility (Union). Figure 2-5 shows the percentage of the portfolio budget, savings, and shareholder incentive claimed in 2015 and planned for 2016 and 2017 for the Large DNV GL 11/10/16 Page 13

29 Volume program. The figure shows that the Large Volume program accounts for a large portion of Union s savings goals at approximately 40%. The Large Volume program did not receive a shareholder incentive in 2015 because it did not meet the minimum scorecard threshold, but it makes up 9% of the total available shareholder incentive in 2016 and Figure Union Large Volume program Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. 2.5 Low income programs Figure 2-6, Figure 2-7, Figure 2-8 show the percentage of the claimed (2015) or planned (2016 and 2017) portfolio savings, claimed (2015) or planned (2016 and 2017) portfolio budget, and claimed (2015) or total available (2016 and 2017) portfolio shareholder incentive for the 2015, 2016, and 2017, Union, and combined low income programs. Participants of a low income natural gas DSM program must meet at least one of the following four requirements: Household income at or below 135% of the most recent Statistics Canada pre-tax Low-Income Cut-Offs for communities of 500,000 or more A recipient of one of the social benefits identified in the DSM Filing Guidelines 5 A participant who resides in social and/or assisted housing, as long as the housing provider is able to provide in writing an indication that their residents are income eligible A household that resides in a community that is targeted for the neighborhood blitz treatment Two programs essentially account for the entire low income portfolio: the single family Low Income Home Winterproofing () and Home Weatherization (Union) programs and the multi-family 5 Ontario Energy Board. Filing Guidelines to the Demand Side Management Framework for Natural Gas Distributors ( ). December 22, Pages 11 and 12. DNV GL 11/10/16 Page 14

30 Low Income Multi-Residential Affordable Housing programs, which are offered by both utilities in all three years ( ). Beginning in 2016, added the Low Income New Construction program and Union added the Furnace End-of-Life program. Union will add the Low Income Aboriginal program in For, there is very little change to the low income budget, savings, and shareholder incentive allocations across 2015 to The single family program is dedicated a larger portion of the budget than the multi-family program, while the multi-family program is projected to achieve more savings. Figure low income programs LI New Construction LI Multi-Res Housing 2017 % Total SI 2017 % Total Savings 2017 % Total Budget 2016 % Total SI 2016 % Total Savings 2016 % Total Budget 2015 % Total SI 2015 % Total Savings 2015 % Total Budget LI Home Winterproofing 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. Where necessary, shareholder incentive and budget are distributed within programs based on the proportion of energy savings for that program. For Union, the single family program is dedicated a larger portion of the budget, although both the single family and multi-family programs projected savings are similar. Compared to, Union s projected savings from its low income programs are considerably smaller. The planned savings for the Low income Furnace End-of-Life in 2016 and 2017 and the Aboriginal program addition in 2017 are negligible. DNV GL 11/10/16 Page 15

31 Figure Union low income programs Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. Where necessary, shareholder incentive and budget are distributed within programs based on the proportion of energy savings for that program. On a combined basis, the low-income programs account for a material portion of the overall portfolio budget (approximately 20%), consistent with the OEB s direction to expand delivery of low income programs across the province. 6 Figure Combined low income programs Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. Where necessary, shareholder incentive and budget are distributed within programs based on the proportion of energy savings for that program. 6 OEB DSM Framework (EB ), December 22, 2014, Section 6.2, Page 26. DNV GL 11/10/16 Page 16

32 2.6 Market transformation and performance-based programs Figure 2-9, Figure 2-10, and Figure 2-11show the percentage of the claimed (2015) or planned (2016 and 2017) portfolio savings, claimed (2015) or planned (2016 and 2017) portfolio budget, and claimed (2015) or total available (2016 and 2017) portfolio shareholder incentive for the 2015, 2016, and 2017, Union, and combined market transformation programs. As a reminder, the market transformation programs do not have savings goals and therefore do not result in lost revenue recovery for the utilities. 7 In 2016, added the Comprehensive Energy Management (CEM), School Energy Competition, and Energy Leaders programs to its suite of market transformation programs. Energy Literacy will be added in In response to the OEB s Decision and Order on s DSM plan, its Home Labelling program concluded in Union added the Run Smart, Strategic Energy Management (SEM), and Commercial Savings by Design programs in Both utilities continued similar residential new construction programs ( s Residential Savings by Design and Union s Optimum Home), while also continues to deliver its Run it Right (RiR) and Commercial Savings by Design programs. s two Savings by Design programs combine to account for the majority its market transformation budget and approximately 10% of the portfolio budget. The larger C&I market transformation programs, RiR and CEM, use approximately 4% of the overall portfolio budget as the second largest market transformation program areas. Figure market transformation programs Home Labelling Energy Literacy Energy Leaders School Energy Competition 2017 % Total SI 2017 % Total Budget 2016 % Total SI 2016 % Total Budget 2015 % Total SI 2015 % Total Budget Comprehensive Energy Management Commercial Savings by Design Run it Right Residential Savings by Design 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. SEM is more similar to Comprehensive Energy Management than Run Smart or Run it Right, but the SEM program budget could not be distinguished from that of the Run Smart program. Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, s Run it Right and Comprehensive Energy Management programs have targets on both the market transformation and resource acquisition scorecards with small energy savings goals. DNV GL 11/10/16 Page 17

33 Similar to, the proportion of Union s market transformation budget allocated to the Optimum Home and Commercial Savings by Design is large. Union s Run Smart and SEM programs are allocated a negligible portion of the overall portfolio budget. Figure Union market transformation programs Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. SEM is more similar to Comprehensive Energy Management than Run Smart or Run it Right, but the SEM program budget could not be distinguished from that of the Run Smart program. Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Combined, the new construction programs are the leading market transformation programs, with approximately 7% of the budget and 6% of the shareholder incentive in 2016 and The C&I performance programs (RiR, CEM, Run Smart, SEM) make up the next largest segment at 3% of the budget and shareholder incentive. School Energy Competition, Energy Leaders, and Energy Literacy make very little impact on budget or shareholder incentive (1%) combined. DNV GL 11/10/16 Page 18

34 Figure Combined market transformation programs Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. SEM is more similar to Comprehensive Energy Management than Run Smart or Run it Right, but the SEM program budget could not be distinguished from that of the Run Smart program. Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, , 2016, and 2017 portfolio by program Table 2-3 (2015), Table 2-4 (2016) and Table 2-5 (2017) repeat the data used to create the distribution of budget, savings, and shareholder incentive figures in the previous sections. The tables are formatted to show the relative percentages for each column. The green column is for budget (, Union, and combined), blue for savings, and yellow for shareholder incentive. Darker cells represent greater percentages; lighter cells represent lesser percentages. The values in the table are the percentages across the entire portfolio. The tables reiterate the expected conclusion from section 2.2 that the resource acquisition programs make up the largest percentage of budget, savings, and shareholder incentive. Table 2-6 shows the year in which each program started, whether it is similar to an existing Ontario electricity conservation and demand management (CDM) program, and whether an independent verification was previously conducted on that program. For the purposes of the EM&V plan, the EC defines independent verification as a third-party verification of gross savings, NTG, or process evaluation. The table shows that most programs are similar to current electricity CDM offerings. Most programs have not been independently verified; those verifications that have been conducted since 2012 relate to custom C&I and large volume projects and some measures in the multi-family market. DNV GL 11/10/16 Page 19

35 Table 2-3. Distribution of 2015 budget, savings, and shareholder incentive across programs for Union,, and combined Union Combined Program Name Resource Acquisition Programs Res Home Energy Conservation/ Reno Rebate Energy Savings Kit C&I Custom C&I Prescriptive Large Volume Program Large Volume Low Income Programs LI Home Winterproofing/ Weatherization LI Multi-Res Housing Market Transformation Programs Residential Savings by Design / Optimum Home Run it Right Commercial Savings by Design Home Labelling Total 2015 % Total Budget 2015 % Total Savings 2015 % Total SI 2015 % Total Budget 2015 % Total Savings 2015 % Total SI 2015 % Total Budget 2015 % Total Savings 2015 % Total SI 34% 12% 13% 10% 4% 7% 23% 7% 10% NA NA NA 10% 1% 1% 5% 1% 1% 20% 64% 42% 24% 43% 44% 22% 51% 43% 6% 12% 8% 13% 12% 11% 9% 12% 9% NA NA NA 10% 37% 0% 4% 24% 0% 16% 3% 5% 21% 2% 12% 18% 3% 8% 8% 8% 12% 9% 1% 17% 8% 4% 14% 7% NA 10% 4% NA 8% 6% NA 9% 4% 0% 0% NA NA NA 2% 0% 0% 3% NA 4% NA NA NA 2% NA 2% 0% NA 6% NA NA NA 0% NA 3% 100% 100% 100% 100% 100% 100% 100% 100% 100% Legend: Low % of portfolio Budget Savings Shareholder Incentive High % of portfolio Low % of portfolio High % of portfolio Low % of portfolio High % of portfolio Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. Where necessary, shareholder incentive and budget are distributed within programs based on the proportion of energy savings for that program. DNV GL 11/10/16 Page 20

36 Table 2-4. Distribution of 2016 budget, savings, and shareholder incentive across programs for Union,, and combined Union Combined Program Name Resource Acquisition Programs Res Home Energy Conservation/ Reno Rebate SmartThermostats C&I Custom C&I Prescriptive C&I Direct Install C&I Small Commercial New Construction Large Volume Program Large Volume Low Income Programs LI Home Winterproofing/ Weatherization LI Multi-Res Housing LI New Construction LI Aboriginal LI Furnace End of Life Market Transformation Programs Residential Savings by Design / Optimum Home Run it Right / Run Smart/SEM Commercial Savings by Design Comprehensive Energy Management School Energy Competition Energy Leaders Energy Literacy Total 2016 % Total Budget 2016 % Total Savings 2016 % Total SI 2016 % Total Budget 2016 % Total Savings 2016 % Total SI 2016 % Total Budget 2016 % Total Savings 2016 % Total SI 27% 10% 22% 20% 4% 18% 24% 6% 20% 2% 2% 2% NA NA NA 1% 1% 1% 16% 58% 30% 21% 35% 31% 18% 43% 31% 5% 14% 7% 18% 13% 11% 11% 13% 9% 11% 6% 3% 1% 0% 0% 7% 2% 2% 1% NA NA NA NA NA 0% 0% 0% NA NA NA 9% 45% 9% 4% 31% 5% 13% 3% 10% 17% 2% 14% 15% 2% 12% 7% 6% 10% 7% 1% 10% 7% 2% 10% 2% 0% 2% NA NA NA 1% 0% 1% NA NA NA NA NA NA 0% 0% 0% NA NA NA 2% 0% 1% 1% 0% 0% 7% NA 3% 2% NA 2% 5% NA 3% 3% 0% 2% 1% 0% 1% 2% 0% 2% 3% NA 3% 1% NA 2% 2% NA 3% 1% 0% 2% NA NA NA 1% 0% 1% 1% NA 1% NA NA NA 0% NA 1% 1% NA 0% NA NA NA 0% NA 0% 0% NA 0% NA NA NA 0% NA 0% 100% 100% 100% 100% 100% 100% 100% 100% 100% Legend: Low % of portfolio Budget Savings Shareholder Incentive High % of portfolio Low % of portfolio High % of portfolio Low % of portfolio High % of portfolio Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, s Run it Right and Comprehensive Energy Management programs have targets on both the market transformation and resource acquisition scorecards with small energy savings goals. SEM is more similar to Comprehensive Energy Management than Run Smart or Run it Right, but the SEM program budget could not be distinguished from that of the Run Smart program. Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. Where necessary, shareholder incentive and budget are distributed within programs based on the proportion of energy savings for that program. DNV GL 11/10/16 Page 21

37 Table 2-5. Distribution of 2017 budget, savings, and shareholder incentive across programs for Union,, and combined Union Combined Program Name Resource Acquisition Programs Res Home Energy Conservation/ Reno Rebate Smart Thermostats C&I Custom C&I Prescriptive C&I Direct Install C&I Small Commercial New Construction Large Volume Program Large Volume Low Income Programs LI Home Winterproofing/ Weatherization LI Multi-Res Affordable Housing LI New Construction LI Aboriginal LI Furnace End of Life Market Transformation Programs Residential Savings by Design / Optimum Home Run it Right / Run Smart/SEM Commercial Savings by Design Comprehensive Energy Management School Energy Competition Energy Leaders Energy Literacy Total 2017 % Total Budget 2017 % Total Savings 2017 % Total SI 2017 % Total Budget 2017 % Total Savings 2017 % Total SI 2017 % Total Budget 2017 % Total Savings 2017 % Total SI 29% 12% 22% 23% 5% 18% 26% 7% 20% 3% 4% 2% NA NA NA 2% 1% 1% 14% 55% 30% 18% 37% 31% 16% 44% 31% 4% 13% 7% 16% 14% 11% 9% 13% 9% 10% 6% 3% 6% 2% 0% 8% 3% 2% 3% 1% 0% NA NA NA 1% 0% 0% NA NA NA 7% 40% 9% 3% 26% 5% 12% 3% 10% 14% 2% 14% 13% 2% 12% 7% 6% 10% 8% 1% 10% 7% 3% 10% 2% 0% 2% NA NA NA 1% 0% 1% NA NA NA 1% 0% 0% 0% 0% 0% NA NA NA 2% 0% 1% 1% 0% 0% 6% NA 3% 2% NA 2% 4% NA 3% 3% 0% 2% 1% 0% 1% 2% 0% 2% 2% NA 3% 2% NA 2% 2% NA 3% 2% 0% 2% NA NA NA 1% 0% 1% 1% NA 1% NA NA NA 1% NA 1% 1% NA 0% NA NA NA 0% NA 0% 1% NA 0% NA NA NA 1% NA 0% 100% 100% 100% 100% 100% 100% 100% 100% 100% Legend: Low % of portfolio Budget Savings Shareholder Incentive High % of portfolio Low % of portfolio High % of portfolio Low % of portfolio High % of portfolio Source: Ontario Energy Board. (2016). Decision and Order EB /EB Gas Distribution, Inc., 2015 Demand Side Management Draft Annual Report, April 22, Union Gas Limited, 2015 Demand Side Management Draft Annual Report, April 22, s Run it Right and Comprehensive Energy Management programs have targets on both the market transformation and resource acquisition scorecards with small energy savings goals. SEM is more similar to Comprehensive Energy Management than Run Smart or Run it Right, but the SEM program budget could not be distinguished from that of the Run Smart program. Total Available Shareholder Incentive is based on 150% achievement of scorecard metrics and 2017 Planned Energy Savings are calculated by a formula based on 100% achievement of the previous year s energy savings goal. Where necessary, shareholder incentive and budget are distributed within programs based on the proportion of energy savings for that program. DNV GL 11/10/16 Page 22

38 Table 2-6. Program maturity, similarity to electricity CDM offerings, and previous evaluations Program Year Started Independently Verified? Similar CDM Program Resource Acquisition Residential Home Energy Conservation / Home Reno Rebate 2012 No Heating and Cooling Incentive Program Residential Adaptive Thermostats 2016 New Program None C&I Custom More than 10 years Yes Retrofit Program Process and Systems C&I Prescriptive More than 10 years No Retrofit Program C&I Direct Install 2016 New Program C&I Small Commercial New Construction 2016 New Program Small Business Lighting High Performance New Construction Large Volume Large Volume 2013 Yes Retrofit Program Process and Systems Low Income LI Home Winterproofing/ Weatherization 2008 (Winterproofing) 2012 (Weatherization) LI Multi-Res Affordable Housing 2012 Yes Limited prescriptive measures Yes (Custom and limited prescriptive measures) Home Assistance Program Retrofit Program (Social Housing Adder) LI New Construction 2016 New Program None LI Aboriginal 2017 Not Launched HONI Aboriginal Program Conservation on the Coast (Five Nations) Custom Home Assistance Program DNV GL 11/10/16 Page 23

39 Program Year Started Independently Verified? Similar CDM Program LI Furnace End of Life 2012 No None Market Transformation Residential Savings by Design / Optimum Home 2012 No New Home Construction Run it Right / Run Smart 2012 (Run it Right) 2016 (Run Smart) New Program Existing Building Commissioning Commercial Savings by Design 2012 () 2016 (Union) No High Performance New Construction Strategic / Comprehensive Energy Management 2016 New Program Provincial Energy Managers Program School Energy Competition 2016 New Program None Energy Leaders Pilot 2016 Not Launched None Energy Literacy Program 2017 New Program None SEM is more similar to Comprehensive Energy Management than Run Smart or Run it Right, but the SEM program budget could not be distinguished from that of the Run Smart program. SEM has more similarity to the Provincial Energy Managers Program than CEM. 2.8 Portfolio summary Based on the information in the previous sections, the EC team suggests the program-level prioritization shown in Table 2-7. The EC recommends concentrating evaluation resources on the resource acquisition programs, especially C&I, which make up the majority of energy savings and are allocated the greatest portion of the shareholder incentive. The EC recommends minimizing spending on market transformation programs with the following exceptions: Conduct at least one comprehensive verification of each market transformation program that has an associated shareholder incentive. Consider the effect of RiR, Run Smart, SEM, and CEM on the verified savings, spillover savings, and free ridership results of the C&I resource acquisition and large volume programs, especially C&I Custom and C&I Prescriptive. Consider multi-year market assessment studies to gauge the success of the program designs and implementation and help guide program spending moving forward. DNV GL 11/10/16 Page 24

40 Table 2-7. EC-Recommended program prioritization: portfolio analysis Program(s) Reason Top Priority Group C&I Custom, Large Volume, and C&I Prescriptive Residential Home Energy Conservation / Home Reno Rebate Low Income Winterproofing / Weatherization Low Income Multi-residential Housing Together, make up more than 80% of program savings and almost half of shareholder incentive. Second-largest in savings and 20% of shareholder incentive Third-largest in shareholder incentive Fourth-largest in shareholder incentive Mid-Level Priority Group Run it Right / Run Smart / Comprehensive Energy Management / SEM C&I Direct Install Residential Savings by Design / Optimum Home Commercial Savings by Design Residential Adaptive Thermostats Measureable contribution to shareholder incentive; likely overlap with C&I Custom and Prescriptive, where largest savings reside Relatively large budget, measureable contribution to savings and shareholder incentive. Measureable contribution to shareholder incentive Measureable contribution to shareholder incentive New program, measureable contribution to savings and shareholder incentive. Lowest Priority Group Energy Leaders / Energy Literacy No savings, no shareholder incentive School Energy Competition C&I Small Commercial New Construction / Low Income New Construction / Low Income Aboriginal Low Income Furnace End of Life No savings, very small shareholder incentive Small savings and shareholder incentive. New programs; suggest delivery review but no impact evaluation. Small savings and shareholder incentive. DNV GL 11/10/16 Page 25

41 PROGRAM EVALUATION PLANS Similar to previous years, all 2015 programs will be verified, with additional evaluation activities on some programs. Table 3-1 summarizes the activities planned for the 2015 program year evaluations. The full memos that discuss the details of these activities are located in APPENDIX B. Table 3-1. Summary of the evaluations of the 2015 program year portfolio Program(s) 2015 Evaluation Activities Rationale Residential Home Energy Conservation () / Home Reno Rebate (Union) C&I Custom (both) Large Volume (Union) C&I Prescriptive (both) Low Income Home Winterproofing () / Weatherization (Union) LI Multi-Res Affordable Housing (both) Residential Savings by Design () / Optimum Home (Union) Commercial Savings by Design () Run it Right () Home Labelling () Energy Savings Kit (Union) Not included in 2015 program year activities Full verification, including a process similar to the previous custom savings verification plus the addition of a NTG study Verification: telephone supplemented by limited on-sites Not included in 2015 program year activities Full verification for custom projects using a process similar to the previous verification Verification: telephone confirmation of scorecard metrics NTG study No activities Will be covered in the plan Verification is similar to what has been done in the past but with a greater level of rigor. NTG study was already in progress Programs have not been independently verified in past years Will be covered in the plan Verification is similar to what has been done in the past but with a greater level of rigor. Programs are eligible for more than 10% of the available shareholder incentive NTG study was already in progress Programs are no longer offered 3.1 Commercial and Industrial Custom, Large Volume, Run it Right, and Low Income Multi-family programs The Commercial and Industrial (C&I) and Large Volume programs will have an integrated evaluation that consists of two activities that have traditionally been treated as unique: gross savings verification and a netto-gross (NTG) study. The Low Income Multi-family program will have a gross savings verification conducted in conjunction with the C&I Custom and Large Volume verification activities. The Run it Right program will be included in the NTG study. DNV GL 11/10/16 Page 26

42 The gross savings verification will follow a process similar to what has been done in the past. A sample of custom projects from the C&I Custom, Large Volume, and Low Income Multi-family programs will be selected for verification. Based on the type and size of the project installed, the verification contractor will perform a telephone-based or on-site verification of the energy savings estimated by the program. The onsite verification activities could fall into multiple rigor categories, with a small number receiving very high rigor on-sites and a larger number receiving standard on-sites. The data collected during the telephone call or on-site visit will be used to produce a verified gross savings estimate for that site. The NTG study was in progress at the time the EC contract was awarded and will largely continue as previously planned. The study will select a sample of projects from the C&I Custom, Large Volume, and Run it Right programs to receive a telephone survey used to determine the level of utility influence on the decision to install the high efficiency projects. Based on the telephone data collected, the sampled projects will be assigned a project-level NTG factor. The gross savings verification sample will be a sub-set of the NTG sample. 3.2 Commercial and Industrial Prescriptive program A gross savings verification study will be used to confirm the energy savings estimated by the C&I Prescriptive program. The verification contractor will use telephone surveys to confirm the type and quantities of the energy efficiency equipment installed, and confirm that the equipment is still operating as planned. The contractor will also confirm that the installed equipment qualified for an incentive according to program rules. 3.3 Residential Savings by Design, Optimum Home, and Commercial Savings by Design programs Telephone verification questionnaires will be used to determine whether the scorecard metrics for the Residential Savings by Design, Optimum Home, and Commercial Savings by Design programs were achieved for the 2015 program year. DNV GL 11/10/16 Page 27

43 AND 2017 PROGRAM EVALUATION PLANS The following sections describe all of the programs included within and Union s DSM portfolios. Each program section includes the following sub-sections: Program sub-section 1: describes the program Program sub-section 2: identifies the key objectives of the program Program sub-section 3: describes in a table the risks associated with the program, the likelihood of that risk occurring, the potential impact of the risk, the evaluation activity that can mitigate the risk, and the relative evaluation cost associated with that activity Program sub-section 4: provides a recommended evaluation activity with a high-level overview of the options and the tasks necessary to complete the activities. 4.1 Home Energy Conservation () Home Reno Rebate (Union) Program Type Relative Size Resource Acquisition 20% of utility shareholder incentive ~25% of portfolio budget ~6% of portfolio savings Offered Since 2012 Past Evaluations Savings have been assessed on a project-by-project basis by third-party certified energy advisors/evaluators using building modeling software. No formal program impact evaluation has been conducted yet. program s spreadsheet was verified in 2014 to validate correct use of assumptions and to ensure deep savings participants met the criteria. *Percentages are of the combined portfolio, inclusive of both utilities Program description The Home Energy Conservation () and Home Reno Rebate (Union) programs offer residential customers 8 home retrofit measures, including thermal insulation, space and water heating system replacements, air sealing, drain water heat recovery, and windows. They are based on Natural Resources Canada (NRCan) s ecoenergy program which existed from 2007 to The programs use a whole-house approach with a special focus on deep savings. Starting in 2016, participants are defined as: Customers that implement at least two major measures, with average annual gas savings of 15% of combined baseline space and water heating consumption across all participants () 8 Rate 1 residential customers () and Residential customers living in detached, semi-detached, townhouses and individually metered row houses (Union). DNV GL 11/10/16 Page 28

44 Customers that implement at least two qualifying measures. In addition, the aggregate of all the homes counting towards the participant scorecard metric must achieve, on average, at least a 15% reduction 9 in annual gas usage as determined through comparing a pre and post energy assessment (Union) targets sections of the residential sector with higher than average consumption, higher household income, dwellings built years ago, above average proportion of single detached dwellings, and low participation in NRCan s ecoenergy program. Union targets detached homes built in 1994 or before with a natural gas heating system. In 2015, the Union program was offered to all residential customers within Central and Southwestern Ontario. Participants must contact a certified energy advisor/evaluator who undertakes pre and post-retrofit home energy evaluations to assess energy savings using building modeling software Key objectives Both programs address key barriers for home energy retrofits: 10 Significant initial investment for homeowners (); High product installation and energy assessment costs (Union) Transactional costs for homeowners (identifying potential measures and prioritizing them) (); Lack of customer awareness regarding opportunities to save energy in their home, particularly when it comes to out of sight measures such as insulation (Union) Finding reliable contractors with knowledge and experience in energy retrofits (); Lack of channel partner expertise in selling the long-term benefits of high efficiency (Union) Each utility has identified additional barriers for their service territory: Lower return on investment for gas measures relative to electricity, in addition to the competitiveness of LDC incentives, that may impact homeowner decision making () Reluctance among channel partners to delay or complicate a sale by referring clients to Service Organizations 11 (Union) Channel capacity: expanding the program requires additional channel and Service Organization relationships (Union) The overall objectives for the programs 12 are: Maximizing cost effective natural gas savings Pursuing deep energy savings Avoiding lost opportunities Maintaining customer satisfaction In summary, the offering theory 13 is as follows: 9 From 25% before Reduced value to account for change in heating system efficiency baseline (at least 90% AFUE for furnaces and 82% AFUE for boilers). EB Exhibit B.T5.Union.CCC Union Gas Limited DSM Plan: EB , Exhibit A, Tab 3, Appendix A, page 7 of Service Organizations are program partners that complete and initial energy assessment and identify energy saving opportunities in the home. Union Gas Limited DSM Plan: EB Exhibit A, Tab 3, Appendix A, page 4 of For Union s program. 13 For Union s program. The program theory for should be similar. DNV GL 11/10/16 Page 29

45 In the short-term, the offering elements will increase the level of customer awareness about the programs offering and their benefits. Service organizations and contractors will promote the offering to prospective participants. Service organizations will increase their capacity to complete assessments and contractors will understand the structure of the offering and be prepared to install measures. Customer awareness and service organization capacity will lead to completed pre-assessments, during which potential measures will be identified. In the medium-term, participants will make the decision to install measures. Contractors will install these measures. In the long-term, the offering will generate natural gas savings and customer satisfaction.14 The scorecard metrics affecting the shareholder incentive are shown in Table 4-1. Table scorecard metrics: Home Energy Conservation and Home Reno Rebate Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Union Gas Cumulative Natural Gas Savings 1,214.1 million CCM/year (all resource acquisition programs) 85.7 million CCM/year (HRR 2016 target) $337,647 Home Reno Rebate Participants 3,300 participants/year $1,593,625 Small Volume Customers Cumulative Natural Gas Savings million CCM/year (all small volume programs) million CCM/year (HEC 2016 target) $960,870 Residential Deep Savings Participants 8,259 participants/year $1,358,500 Source: EB / EB , Schedule B. Where applicable, shareholder incentives based on savings target of several programs have been prorated to specific programs using 2016 approved CCM targets of Schedule A Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate and shareholder incentive amount. Table 4-2 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. 14 Theory is drafted as per a pure resource acquisition program, although this type of program can bring market transformation outcomes as well (e.g., larger and more skilled contractor base). DNV GL 11/10/16 Page 30

46 Table 4-2. Risks, impacts, and evaluation activities: Home Energy Conservation and Home Reno Rebate Risk Description Likelihood Impact Possible Evaluation Activity(ies) Relative Evaluation Cost Both Programs Jointly Free ridership assumptions may be off (15% on wholehome basis) Pre-retrofit building characteristics, especially hidden features (e.g., wall insulation) are difficult to assess and may induce overor under- estimation of savings Assumptions of base load and occupancy for internal gains may be different from actual participant characteristics Other modeling accuracy issues, modeled pre-retrofit consumption is not calibrated with actual consumption Measures might not be properly installed, especially for do-it-yourself jobs, which would affect performance Rebound effect and other behavioral changes not being captured High High Participant survey, interviews with trade allies High High Billing analysis, desk review of modeling files, site visits High Low Billing analysis, desk review of modeling files, participant survey Medium Medium Billing analysis, Comparison of preretrofit consumption and modeled preretrofit consumption High High Billing analysis (especially with doit-yourself jobs subgroup), On-site inspections Low Low Billing analysis, Participant survey Low Medium (billing analysis, site visits), Low (desk review) Medium (billing analysis), Low (desk review, participant survey) Medium (billing analysis), Low (real vs modeled pre-retrofit comparison) Medium Medium (billing analysis), Low (participant survey) DNV GL 11/10/16 Page 31

47 Risk Description Likelihood Impact Possible Evaluation Activity(ies) Relative Evaluation Cost Energuide rating system and modeling software update Measures EUL assumptions being different from real useful life, affecting costeffectiveness and cumulative savings estimates Measures incremental cost assumptions being different from real incremental costs, affecting cost-effectiveness results Measures non-energy benefit assumptions being different from real non-energy benefit effects Union Gas Only Baseline assumptions for furnace replacements Only Number of deep savings might be different following evaluation of savings Medium Low Extra quality control on files during the transition period Medium High Persistence study, Review of EUL assumptions High High Market study, Review of cost assumptions High Medium Participant survey Market study Low Medium Baseline study, Review of modeling files, Participant survey (for early replacement vs ROB furnaces) High High Review of deep savings estimate Low High (persistence study), Low (review of assumptions) Medium (market study), Low (review of assumptions) Low (participant survey), Medium (market study) Medium (baseline study), Low (review of files, participant survey) Low DNV GL 11/10/16 Page 32

48 Risk Description Likelihood Impact Possible Evaluation Activity(ies) Relative Evaluation Cost Baseline assumptions for furnace replacements High High Baseline study, Review of modeling files, Participant survey (for early repl. vs ROB furnaces) Medium (baseline study), Low (review of files, participant survey) Natural Resources Canada is in the process of updating the EnerGuide Rating System for homes, including updated energy modelling software, house evaluation procedures and quality assurance procedures. The new rating system could be implemented during For all HOT2000 files, Union uses a minimum AFUE of 90% for furnaces and 82% for boilers starting Source : EB , Union Exhibit A, Tab 3, Appendix C, p. 10. Although Union also calculates the number of deep savings retrofits, this metric is not used for shareholder incentive determination Evaluation activity recommendation This section describes the potential evaluation activities in greater detail with a description of the study objectives and a high-level outline of the key tasks. Although the evaluation activities are outlined as discrete tasks, they can be combined to produce more comprehensive studies. For example, the billing analysis could be supplemented with participant and trade ally surveys to produce net savings in addition to gross savings from the billing analysis. For these programs, the EC team recommends that a billing analysis be initiated as soon as possible. The billing analysis should include desk reviews to identify the cause of any discrepancies between the results of the billing analysis and the program-estimated savings through the building simulations. It should also include participant surveys to produce a net-to-gross adjustment factor and verify occupancy assumptions in the building model. In 2017, we recommend initiating a market assessment to look at things such as preretrofit building characteristics, base load assumptions, and baseline furnace assumptions which will improve the building simulation results. These activities (and other options) are described in greater detail in the following sections Billing analysis Program gross savings estimates are currently based on building energy modeling. A billing analysis can verify average gross savings relative to the existing equipment baseline, for all program households, using program tracking data and historic billing consumption data. This evaluation activity addresses these risks from Table 4-2: Poorly assessed pre-retrofit building characteristics: The program simulations may be using general, erroneous, or unverified pre-retrofit condition assumptions. Incorrect building characteristics in the pre- or post-retrofit simulation will result in inaccurate savings estimates. The billing analysis will identify the true savings relative to an existing equipment baseline. Base load and occupancy assumptions: Incorrect base load or occupancy assumptions will also result in inaccurate savings estimates. The billing analysis will identify the true savings relative to an existing equipment baseline. DNV GL 11/10/16 Page 33

49 Uncalibrated simulations: Calibrating the simulations with billing data will increase the accuracy of the savings estimate. The billing analysis will identify the true savings relative to an existing equipment baseline. Improper installation: The billing analysis will identify the true savings relative to the actual installation method, not an ideal. Rebound effect and other behavioral changes: the billing analysis will identify the true savings relative to the actual equipment usage in the house, not an ideal. The objectives of the billing analysis would be to: Determine the energy savings (using an existing equipment baseline) resulting from program participation through a difference-in-difference econometrics modeling approach on the population of program participants. Adjust savings for a standard efficiency baseline where appropriate, such as when a high efficiency furnace is installed to replace a failed furnace. Determine a factor that can be used to adjust the simulation savings going forward to reflect the savings one might get from a billing analysis. The billing analysis should include the following tasks: Request, receive, clean, and assemble billing data, program data, and energy modeling files, from and Union, for all program participants. Estimate site-level, weather-normalized models for pre- and post-installation periods for program participants. Construct a comparison group to address non-program exogenous change. Determine the gross average savings per household resulting from program participation. Compare the billing analysis result to the building simulation result and identify the source of the differences, using other evaluation means such as a participant survey, and a desk review of modeling files. Calculate an adjustment factor that can be used to correct the simulation estimates going forward. Write a report on the methodology, billing analysis results, comparison results, and recommendations that may help improve program design Participant survey and trade ally interviews Program free ridership and spillover are currently based on stipulated values. In addition, the building simulations use assumptions regarding occupancy, base load, and participant behavior that may not reflect the participant population. Finally, the baseline assumption for furnaces may or may not accurately reflect the number of early replacements versus those that are replaced on burnout. A participant survey can address these issues and investigate customer and trade ally satisfaction with the program offerings. This evaluation activity addresses these risks from Table 4-2: Free ridership assumptions: a well-crafted participant and trade ally survey can collect the data necessary to estimate program free ridership. Additional questions can be added to estimate spillover. Base load and occupancy assumptions: A survey can be completed with a sample of participants to verify the input assumptions. Rebound effect and other behavioral changes: A survey can ask customers to identify how they have changed behaviors and energy usage since the measures were installed. The survey would be DNV GL 11/10/16 Page 34

50 conducted with a sample of participants and a control group of non-participants to correct for exogenous trends. Furnace replacement assumptions: A survey can identify whether an installed furnace (or other equipment) was replace-on-burnout or early replacement. Non-energy benefit (NEB) assumptions: A survey can ask participants about the non-energy benefits resulting from measure installation. The objectives of participant survey would be to: Determine the free ridership rate and investigate evidence of participant spillover Verify the building simulation assumptions used for base load, occupancy rate, set points, etc. Identify behavioral changes, including any rebound effect, that would have an impact on the energy savings estimate Determine the rate of furnace change-outs that are early replacement versus replace on burnout Confirm customer and trade ally satisfaction with program offerings and identify potential improvements Gather evidence of NEBs resulting from measure installation The participant survey should include the following tasks: Design a sample that represents the program population, oversampling furnace replacements Write and deliver a survey to address the research questions. Analyze the survey data to determine free ridership, simulation assumptions, behavioral changes, rate of early replacement, evidence of NEBs, and satisfaction. Write a report on the methodology, results, and recommendations that may help improve program design Market studies Program measure life, furnace baseline, and incremental costs are currently based on stipulated values. One or more market studies can gather data that more directly reflect the participants of the programs. This evaluation activity addresses these risks from Table 4-2: Incorrect measure life: A study that assesses effective useful life (EUL) using client surveys and site visits will quantify persistence and degradation of upgrades and savings. Alternatively, EULs can be validated at lower cost using secondary sources, such as other persistence studies or manufacturer data. Inaccurate incremental cost: A market study that identifies the incremental cost of an energy efficient installation relative to a standard efficiency installation in Ontario will result in more accurate cost-benefit study results. Furnace baseline assumptions: A market study that identifies the average installed furnace efficiency in Ontario will more accurately reflect the degree to which the program influences a change. The average installed efficiency may not be the same as the minimum efficiency available. Inaccurate NEB assumptions: A market study that researches and quantifies the financial equivalent of the NEB related to measure installation will result in a more accurate representation of the benefits resulting from the program. The objectives of the market studies would be to: Improve the cumulative savings by more accurately determining measure life Improve the cost-benefit analysis by more accurately determining incremental cost. DNV GL 11/10/16 Page 35

51 Identify the actual average furnace efficiency sold in the Ontario market to improve the building simulation results. Improve the quantification of measure installation by quantifying the NEBs received by participants. The market study should include the following tasks: Investigate the potential data available for study: sales data, government data, program data. Write a work plan to clearly identify the boundaries of the study: what technologies will be studied, what portions of the market will be measured, what will the outcome be. Collect and analyze data according to the work plan. Write a report on the methodology, results, and recommendations that may help improve program design Other options The EC team has identified activities that could replace some of those outlined above for a different cost. For example, a literature review could be used in place of a participant survey to update the occupancy and base load assumptions for the building simulation. This section outlines the other options and where they can be implemented. Desk review of files This evaluation activity addresses these risks from Table 4-2: Base load and occupancy assumptions: A review of existing literature could provide an updated estimate for the building simulations. Poorly assessed pre-retrofit building characteristics: The program simulations may be using general, erroneous, or unverified pre-retrofit condition assumptions. The pre-retrofit (and post-retrofit) building model assumptions can be reviewed to determine whether they are reasonable. The desk review would identify locations with large discrepancies between the billing and simulation results and review the simulation s thermal loads by components to identify anomalous assumptions. Furnace replacement assumptions: The pre- and post-retrofit building model assumptions can be reviewed to determine whether they are reasonable, and if they follow program procedure where applicable. Achievability of deep retrofit savings: If the billing analysis produces an adjustment factor that would result in lower savings percentages than originally estimated by the program, it may be more difficult to achieve the 15% deep savings necessary to reach the shareholder incentive. A desk review could reveal whether the goal should be changed as a result. The objective of the desk review would be to provide a third-party verification of the assumptions used in the energy models in a cost-effective manner, especially if the billing analysis results are not available. Desk reviews could also enhance the billing analysis by comparing the billing results to building simulations in an attempt to identify what caused the differences. The results of the desk review can also be used to improve the assumptions used for the building simulation inputs. Site visits This evaluation activity addresses these risks from Table 4-2: DNV GL 11/10/16 Page 36

52 Poorly assessed pre-retrofit building characteristics: The program simulations may be using general, erroneous, or unverified pre-retrofit condition assumptions. Using technology such as infrared cameras, hidden pre-retrofit building characteristics might be more accurately measured. Improper installation: A site visit could inspect the work that was completed to ensure it was installed properly. The objective of the site visits would be to provide a third-party verification of the assumptions used in the energy models and confirm proper installation of the energy efficiency measures. Site reviews could also enhance the billing analysis by confirming or eliminating the source of discrepancies between the billing analysis and model results. To ensure that the site visits are effective, the pre-retrofit characteristics must be conducted on pre-retrofit sections of the participant building, or on future participant buildings prior to retrofit. 4.2 Residential Adaptive Thermostats () Program Type Relative Size* Resource Acquisition 1% of utility shareholder incentive ~1% of portfolio budget 1% of portfolio savings Offered Since 2016 Past Evaluations No evaluation activities (new program) *Percentages are of the combined portfolio, inclusive of both utilities Program description This program offers adaptive (or smart) thermostats as a stand-alone prescriptive measure to Rate 1 residential customers, especially targeting customers that do not want or need a complete whole-home energy evaluation and retrofit. Residential Adaptive thermostats can anticipate needs, patterns, and behaviors, and can be accessible almost anywhere. They can track whether and where customers are in their home, or in their city, and make modifications to ensure they are comfortable when and where needed. Beyond comfort, these new technologies enable energy savings by either adapting to customer behavior, or through geo-coded data, identifying when occupants are, or are not, home. Smart- or geo-fencing thermostats typically have the following key features and benefits: Easily created schedules Intuitive set up, typically using narrative and lifestyle related questions Proactive or forced automatic energy savings adjustment features Greater control with remote web or app based control Maintenance alerts Ongoing learning of lifestyle schedules and preferences. DNV GL 11/10/16 Page 37

53 has been engaged in discussions with a few LDCs on collaboration around smart thermostats, and will continue to explore design and deployment integration Key objectives The goal of this program is to broadly reach the mass market with a straightforward prescriptive offer that can help customers install smart thermostats and achieve gas savings. Barriers that interrupt customer uptake of smart thermostats typically include: Cost of the unit. The typical cost of an smart thermostat is approximately $250 to $325, whereas programmable thermostats range in cost from $50 to $100. Unfamiliar technology. Wi-Fi technology adoption has been growing in certain areas, and for certain demographics, but has seen a slower adoption curve in other areas and demographics. More features can be perceived as complicated, rather than better. Big brother perception. Customers continue to remain cautious around Wi-Fi connected purchases with concerns around personal data ownership and security. The scorecard metrics affecting the shareholder incentive are shown in Table 4-3. Table scorecard metrics: Residential Adaptive Thermostats Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Small Volume Customers Cumulative Natural Gas Savings million CCM (all small volume programs), 26.3 million CCM (Residential Adaptive Thermostats 2016 target) $223,450 Source: EB / EB , Schedule B. Shareholder incentive based on savings target of several programs has been prorated to specific program using 2016 approved CCM targets of Schedule A Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate and shareholder incentive amount. Table 4-4 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. DNV GL 11/10/16 Page 38

54 Table 4-4. Risks, impacts, and evaluation activities: Residential Adaptive Thermostats Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost New program, no evaluation has been conducted yet Baseline heating/cooling load and set points assumptions might be different than real-life participants baseline conditions Behavior/programming issues: are the thermostat functionalities properly used to maximize savings? High Medium Conduct first evaluation, including process evaluation High High Billing analysis, Participant tracking data / survey Medium Medium Billing analysis, Participant survey, Online data Medium Medium (billing analysis), Low (survey/data) Medium (billing analysis), Low (survey/data) Free ridership higher than expected (current assumption is 4%)* High High Participant survey Low Measures non-energy benefit assumptions being different from real non-energy benefit effects Measure EUL assumptions different from real useful life, affecting costeffectiveness and cumulative savings estimates High Medium Participant survey Market study Medium High Persistence study, Review of EUL assumptions Low (participant survey), Medium (market study) High (persistence study), Low (review of assumptions) DNV GL 11/10/16 Page 39

55 Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Measure incremental cost assumptions different from real incremental costs, affecting costeffectiveness results Low Low Market study, Review of cost assumptions Medium (market study), Low (review of assumptions) *EB New and Updated DSM Measures - Joint Submission from Union Gas Ltd. and Gas Distribution (Exhibit B, Tab 1, Schedule 2, p. 1 of 19) Evaluation activity recommendation This section describes the potential evaluation activities in greater detail with a description of the study objectives and a high-level outline of the key tasks. Although the evaluation activities are outlined as discrete tasks, they can be combined to produce more comprehensive studies. For this program, the EC team recommends that a billing analysis be initiated when participation levels are sufficient to ensure rigor of results, which should be around 1,000 installed thermostats for at least summer, winter, and one shoulder season. The billing analysis should be supplemented with a review of online thermostat data, if available. It should also include participant surveys to produce a net-to-gross adjustment factor and verify pre-installation thermostat type and household behavior. In 2017, we recommend initiating a market assessment to look at incremental cost and measure life. These activities (and other options) are described in greater detail in the following sections Billing analysis A billing analysis would estimate average gross savings, relative to the existing equipment and set points baseline, for all program households, using program tracking data and historical billing consumption data. This evaluation activity addresses these risks from Table 4-4: Baseline conditions: the billing analysis will identify the true savings relative to baseline assumptions that might be different than real-life participants baseline conditions. A survey can also be completed with a sample of participants to verify the assumptions (assuming these info are not tracked by the program). Proper performance of smart thermostats: the billing analysis will identify the true savings relative to an existing equipment baseline, and capture discrepancies in savings if some thermostats were not performing as expected. Proper use of thermostat features: the billing analysis will identify the true savings relative to an existing equipment baseline. Results will reflect any discrepancy between assumptions and real use of thermostats functionalities. A survey and/or analysis of online thermostat data (if available) can also be completed to verify those assumptions. The objectives of the billing analysis would be to: Determine the energy savings (using an existing equipment baseline) resulting from program participation through a difference-in-difference econometrics modeling approach on the population of program participants. DNV GL 11/10/16 Page 40

56 Determine a factor that can be used to adjust program savings going forward to reflect the savings one might get from a billing analysis. The billing analysis should include the following tasks: Request, receive, clean, and assemble billing and program data from for all program participants. Estimate site-level, weather-normalized models for pre- and post-installation periods for program participants. Construct a comparison group to address non-program exogenous change. Determine the gross average savings per household resulting from program participation. Estimate a new deemed savings value to use for smart thermostats going forward. Write a report on the methodology, billing analysis results, comparison results, and recommendations that may help improve program design Participant survey A survey would be conducted with a sample of program participants to collect information on baseline and post-installation set points and the use of thermostat functionalities and to validate assumptions and savings estimates. Free ridership, which is currently assumed to be low (4%), will also be assessed. In addition to the impact evaluation, the survey can also help inform the process evaluation. This part of the evaluation could include trade allies as well. This evaluation activity addresses these risks from Table 4-4: Free ridership assumptions: a well-crafted participant and trade ally survey can collect the data necessary to estimate program free ridership. Additional questions can be added to estimate spillover. Baseline thermostat and heating/cooling set points assumptions: A survey can be completed with a sample of participants to verify baseline assumptions. Behavior/programming issues: A survey can ask customers to verify the actual usage of smart thermostat functionalities. The objectives of participant survey would be to: Determine the free ridership rate Verify baseline conditions Determine post-installation usage patterns Confirm customer (and potentially trade ally) satisfaction with program offerings and identify potential improvements The participant survey should include the following tasks: Design a sample that represents the program population Write and deliver a survey to address the research questions. Analyze the survey data to determine free ridership, baseline conditions, post-installation usage patterns, and satisfaction. Write a report on the methodology, results, and recommendations that may help improve program design. DNV GL 11/10/16 Page 41

57 Market studies Program measure life and incremental costs are currently based on stipulated values. A market study can gather data that more directly reflects the participants of the programs. This evaluation activity addresses these risks from Table 4-4: Incorrect measure life: A market study will identify the expected life of the equipment. A persistence study will assess measures effective useful life using client surveys and site visits to quantify persistence and degradation of upgrades and savings. Alternatively, EULs can be validated at lower cost using secondary sources (other persistence studies, manufacturers data, etc.). Inaccurate incremental cost: A market study that identifies the incremental cost of smart thermostats in Ontario will result in more accurate cost-benefit study results. The objectives of the market studies would be to: Improve the cumulative savings by more accurately determining measure life Improve the cost-benefit analysis by more accurately determining incremental cost. The market study should include the following tasks: Investigate the potential data available for study: sales data, government data, program data. Write a work plan to clearly identify the boundaries of the study: what technologies will be studied, what portions of the market will be measured, what will the outcome be. Collect and analyze data according to the work plan. Write a report on the methodology, results, and recommendations that may help improve program design. 4.3 Home Winterproofing () Home Weatherization, Furnace End-of-Life, and Aboriginal Offerings (Union) Program Type Relative Size Offered Since Past Evaluations Resource Acquisition 12% of utility shareholder incentive ~15% of portfolio budget 2% of portfolio savings 2012 ( implemented a low-income weatherization program during Current Part 3 and Part 9 programs were implemented in 2012) Basic measures have been verified by a third-party evaluator in 2012 for Union s previous program ( Helping Homes Conserve ). Verification covered measure installation, continued use of measure, percent of showers using efficient showerhead, and water heater type. program s spreadsheet was verified in 2014 to validate data entry. *Percentages are of the combined portfolio, inclusive of both utilities. Part 3 references buildings covered by Part 3 of the Ontario Building Code, defined as those exceeding 600 square meters in area or greater than three storeys in height; for residential energy efficiency programs, these are typically multifamily buildings. Part 9 buildings are those with three or fewer storeys and having a building area not exceeding 600 square meters; for residential energy efficiency programs, these are typically single family buildings. DNV GL 11/10/16 Page 42

58 4.3.1 Program description The Home Winterproofing offer (), previously known as Home Weatherization, and The Low Income Home Weatherization and Furnace End-of-Life offerings (Union) aim to reduce energy costs for Part 9 lowincome households by increasing the energy efficiency of their homes, while addressing comfort and some health and safety matters within the homes. Customers can take advantage of free installation of air sealing, attic, wall, and basement insulation, as well as an incentive for a furnace upgrade for end-of-life units. Participants to Union s program also receive a carbon monoxide detector, and installation of up to two energy efficient showerheads, two meters of pipe wrap and a programmable thermostat. Kitchen and bathroom aerators are left behind for self-installation. They can also receive a Health and Safety incentive to remedy qualifying health and safety issues that may impede installation of measures. The incentive level varies by home, as it is dependent on the overall cost-effectiveness of a given project, and are outlined in the Health & Safety Policy that is provided to the contracted Delivery Agent. provides a carbon monoxide monitor where one is not present in the home. At the time of assessment, the home is also prequalified for water conservation measures (e.g., showerheads and aerators) as well as a programmable thermostat, heat reflectors and a drain-water heat recovery unit. While not part of the current program, may include a modest furnace replacement component in the future. customers must address health and safety issues such as asbestos or structural problems before being able to participate in the Home Winterproofing program. The target market for Home Weatherization and Furnace End-of-Life offerings (Union) are customers living in detached, semi-detached, townhouses and row houses, either as part of the private market or within social and assisted housing. The target markets for Home Winterproofing () are Social housing and assisted housing, and income qualified customers residing in low-rise buildings (Ontario Building Code (OBC) Part 9). These programs reach out to low-income households that are often on fixed incomes, rely on income assistance programs, and are generally the most vulnerable to volatile changes in energy markets. Starting in 2017, Union will deploy the Aboriginal offering, a targeted delivery of the Home Weatherization and Furnace End-of-Life offerings to the First Nations community. There are 13 First Nations reserves with residential gas service in Union s franchise area. Select reserves will be targeted each year based on the following criteria: band council election dates; time required to engage with the community, proximity and existing relationships amongst reserves, and Union s internal constraints Key objectives Low income households are difficult to find, primarily because they are unlikely to self-identify as low income. Further, once found, they are hard to reach due to other barriers such as communication and language challenges. Other barriers to participation in energy efficiency programs among this customer segment include: Affordability and access to funds to make equipment upgrades and repairs ( and Union) Awareness or limited understanding of energy and energy use () Competing priorities for basic necessities such as food, shelter, health and safety () Lack of trust nothing is free mentality, or fear of claw backs of government financial assistance ( and Union) Health and safety requirements that must be addressed prior to program participation (Union) Awareness of the offering within the private market (Union) DNV GL 11/10/16 Page 43

59 Communication restrictions that require more targeted materials The overall objectives for the low income offerings are: Reducing energy costs for low income customers ( and Union) Providing non-energy benefits (e.g., improved indoor home comfort and health and safety) for low income customers ( and Union) Customer satisfaction (Union) In summary, the offering theory is as follows: 15 In the short-term, the offering elements will increase the level of customer awareness about Home Weatherization and Furnace End-of-Life and their benefits. Associations and community organizations will help to promote the offering to prospective participants. Customer care agents will help to identify potential participants. In the medium-term, customer awareness will lead to completed pre-assessments, during which participant eligibility will be identified, and potential measures will be identified. Participants will make the decision to install measures. Contractors will install these measures. In the long-term, the offering will generate natural gas savings and customer satisfaction. The scorecard metrics affecting the shareholder incentive are shown in Table 4-5. Table scorecard metrics: Home Winterproofing, Home Weatherization, Furnace End-of- Life, and Aboriginal Offerings Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Union Gas Cumulative Natural Gas Savings 37.8 million CCM $1,567,500 Single Family Cumulative Natural Gas Savings 31.8 million CCM $1,081,575 Source: EB / EB , Schedule B Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate and shareholder incentive amount. Table 4-6 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. 15 For Union s program. The program theory for should be similar. DNV GL 11/10/16 Page 44

60 Table 4-6. Risks, impacts, and evaluation activities: Home Winterproofing, Home Weatherization, Furnace End-of-Life, and Aboriginal Offerings Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Both Programs Jointly Pre-retrofit building characteristics, especially hidden features (e.g., wall insulation) are difficult to assess and may induce over- or underestimation of savings Assumptions of base load and occupancy for internal gains may be different from actual participant characteristics Other modeling accuracy issues, modeled pre-retrofit consumption is not calibrated with actual consumption Measures might not be properly installed (low risk in low income program since utilities have a better control on measure installation) Rebound effect and other behavioral changes not being captured in the net-togross factor High High Billing analysis, Desk review of modeling files, Site visits High Low Billing analysis, Desk review of modeling files, Participant survey Medium Medium Billing analysis, Comparison of preretrofit consumption and modeled preretrofit consumption Low Low Billing analysis, On-site inspections Low Low Billing analysis, Participant survey Medium (billing analysis, site visits), Low (desk review) Medium (billing analysis), Low (desk review, participant survey) Medium (billing analysis), Low (real vs modeled pre-retrofit comparison) Medium Medium (billing analysis), Low (participant survey) DNV GL 11/10/16 Page 45

61 Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Free ridership assumptions may be off (0%, 1% Union)* Energuide rating system and modeling software update Savings assumptions of prescriptive measures Measures EUL assumptions being different from real useful life, affecting cost-effectiveness and cumulative savings estimates Low Low Participant survey, Interviews with trade allies Medium Low Extra quality control on files during the transition period Medium Low Review of savings assumptions Medium Medium Persistence study, Review of EUL assumptions Low Low Low High (persistence study), Low (review of assumptions) Union Gas Only Baseline assumptions for furnace replacements Low Medium Baseline study, Review of modeling files, Participant survey (for early replacement vs ROB furnaces) Medium (baseline study), Low (review of files, participant survey) Natural Resources Canada is in the process of updating the EnerGuide Rating System for homes, including updated energy modelling software, house evaluation procedures and quality assurance procedures. The new rating system could be implemented during *EB New and Updated DSM Measures - Joint Submission from Union Gas Ltd. and Gas Distribution (Exhibit B, Tab 1, Schedule 2, p. 2-3 of 19) Evaluation activity recommendation This section describes the potential evaluation activities in greater detail with a description of the study objectives and a high-level outline of the key tasks. Although the evaluation activities are outlined as discrete tasks, they can be combined to produce more comprehensive studies. For example, the billing analysis could be supplemented with participant and trade ally surveys to produce net savings instead of just gross savings. For these programs, the EC team recommends that a billing analysis be initiated as soon as possible, conducted in conjunction with the evaluation activities undertaken for the Home Conservation and Home DNV GL 11/10/16 Page 46

62 Reno Rebate programs. The billing analysis should include desk reviews to identify the cause of any discrepancies between the results of the billing analysis and the program-estimated savings through the building simulations. It should also include participant surveys to verify occupancy assumptions in the building model and produce a net-to-gross adjustment factor. In 2017, we recommend initiating a market assessment to look at things such as pre-retrofit building characteristics, base load assumptions, and baseline furnace assumptions which will improve the building simulation results. These activities (and other options) are described in greater detail in the following sections Billing analysis Program gross savings estimates are currently based on building energy modeling. A billing analysis can verify average gross savings relative to the existing equipment baseline, for all program households, using program tracking data and historical billing consumption data. This evaluation activity addresses these risks from Table 4-6: Poorly assessed pre-retrofit building characteristics: Incorrect building characteristics in the preor post-retrofit simulation will result in inaccurate savings estimates. The billing analysis will identify the true savings relative to an existing equipment baseline. Base load and occupancy assumptions: Incorrect base load or occupancy assumptions will also result in inaccurate savings estimates. The billing analysis will identify the true savings relative to an existing equipment baseline. Uncalibrated simulations: Calibrating the simulations with billing data will increase the accuracy of the savings estimate. The billing analysis will identify the true savings relative to an existing equipment baseline. Improper installation: The billing analysis will identify the true savings relative to the actual installation method, not an ideal. Rebound effect and other behavioral changes: the billing analysis will identify the true savings relative to the actual equipment usage in the house, not an ideal. The objectives of the billing analysis would be to: Determine the energy savings (using an existing equipment baseline) resulting from program participation through a difference-in-difference econometrics modeling approach on the population of program participants. Adjust savings for a standard efficiency baseline where appropriate, such as when a high efficiency furnace or water heater were installed where the existing equipment would have been replaced anyway without the program (replacements on burnout). Adjust savings using the existing equipment where appropriate, such as when a high efficiency furnace or water heater were installed where no replacement would have occurred without the program (early replacements). Determine a factor that can be used to adjust the simulation savings going forward to reflect the savings one might get from a billing analysis. The billing analysis should include the following tasks: Request, receive, clean, and assemble billing data, program data, and energy modeling files, from and Union, for all program participants. Estimate site-level, weather-normalized models for pre- and post-installation periods for program participants. Construct a comparison group to address non-program exogenous change. DNV GL 11/10/16 Page 47

63 Determine the gross average savings per household resulting from program participation. Compare the billing analysis result to the building simulation result and identify the source of the differences, using other evaluation means such as a participant survey, and a desk review of modeling files. Calculate an adjustment factor that can be used to correct the simulation estimates going forward. Write a report on the methodology, billing analysis results, comparison results, and recommendations that may help improve program design Participant survey and trade ally interviews Program free ridership and spillover are currently based on stipulated values. In addition, the building simulations use assumptions regarding occupancy, base load, and participant behavior that may not reflect the participant population. Finally, the baseline assumption for furnaces may or may not accurately reflect the number of early replacements versus those that are replaced on burnout. A participant survey can address these issues and investigate customer and trade ally satisfaction with the program offerings. This evaluation activity addresses these risks from Table 4-6: Freeridership assumptions: a well-crafted participant and trade ally survey can collect the data necessary to estimate program free ridership. While this effect is presumably very low or nonexistent in a low income program, a survey will serve to validate program assumptions. Additional questions can be added to estimate spillover. Base load and occupancy assumptions: A survey can be completed with a sample of participants to verify the input assumptions. Rebound effect and other behavioral changes: A survey can ask customers to identify how they have changed behaviors and energy usage since the measures were installed. The survey would be conducted with a sample of participants and a control group of non-participants to correct for exogenous trends. Furnace replacement assumptions: A survey can identify whether an installed furnace (or other equipment) was replace-on-burnout or early replacement. Non-energy benefit (NEB) assumptions: A survey can ask participants about the non-energy benefits resulting from measure installation. The objectives of participant survey would be to: Determine the free ridership rate. Verify the building simulation assumptions used for base load, occupancy rate, set points, etc. Identify behavioral changes, including any rebound effect that would have an impact on the energy savings estimate. Determine the rate of furnace change-outs that are early replacement versus replace on burnout. Confirm customer and trade ally satisfaction with program offerings and identify potential improvements. Gather evidence of NEBs resulting from measure installation The participant survey should include the following tasks: Design a sample that represents the program population, oversampling furnace replacements. Write and deliver a survey to address the research questions. Analyze the survey data to determine free ridership, simulation assumptions, behavioral changes, rate of early replacement, evidence of NEBs, and satisfaction. DNV GL 11/10/16 Page 48

64 Write a report on the methodology, results, and recommendations that may help improve program design Market studies Program measure life and furnace baseline are currently based on stipulated values. One or more market studies can gather data that more directly reflects the participants of the programs. This evaluation activity addresses these risks from Table 4-6: Incorrect measure life: A study that assesses effective useful life (EUL) using client surveys and site visits will quantify persistence and degradation of upgrades and savings. Alternatively, EULs can be validated at lower cost using secondary sources, such as other persistence studies or manufacturers data. Furnace baseline assumptions: A market study that identifies the average installed furnace efficiency in Ontario will more accurately reflect the degree to which the program influences a change. The average installed efficiency may not be the same as the minimum efficiency available. Inaccurate NEB assumptions: A market study that researches and quantifies the financial equivalent of the NEB related to measure installation will result in a more accurate representation of the benefits resulting from the program. The objectives of the market studies would be to: Improve the cumulative savings by more accurately determining measure life. Identify the actual average furnace efficiency sold in the Ontario market to improve the building simulation results. Improve the quantification of measure installation by quantifying the NEBs received by participants. The market study should include the following tasks: Investigate the potential data available for study: sales data, government data, program data. Write a work plan to clearly identify the boundaries of the study: what technologies will be studied, what portions of the market will be measured, what will the outcome be. Collect and analyze data according to the work plan. Write a report on the methodology, results, and recommendations that may help improve program design Other options The EC team has identified activities that could replace some of those outlined above for a different cost. For example, a literature review could be used in place of a participant survey to update the occupancy and base load assumptions for the building simulation. This section outlines the other options and where they can be implemented. Desk review of files This evaluation activity addresses these risks from Table 4-6: Base load and occupancy assumptions: A review of existing literature could provide an updated estimate for the building simulations. Poorly assessed pre-retrofit building characteristics: The pre-retrofit (and post-retrofit) building model assumptions can be reviewed to determine whether they are reasonable. The desk review would DNV GL 11/10/16 Page 49

65 identify locations with large discrepancies between the billing and simulation results and review the simulation s thermal loads by components to identify anomalous assumptions. Furnace replacement assumptions: The pre- and post-retrofit building model assumptions can be reviewed to determine whether they are reasonable, and if they follow program procedure where applicable. The objective of the desk review would be to provide a third-party verification of the assumptions used in the energy models in a cost-effective manner, especially if the billing analysis results are not available. Desk reviews could also enhance the billing analysis by comparing the billing results to building simulations in an attempt to identify what caused the differences. The results of the desk review can also be used to improve the assumptions used for the building simulation inputs. Site visits This evaluation activity addresses these risks from Table 4-6: Poorly assessed pre-retrofit building characteristics: Using technology such as infrared cameras, hidden pre-retrofit building characteristics might be more accurately measured. Improper installation: A site visit could inspect the work that was completed to ensure it was installed properly. The objective of the site visits would be to provide a third-party verification of the assumptions used in the energy models and confirm proper installation of the energy efficiency measures. Site reviews could also enhance the billing analysis by confirming or eliminating the source of discrepancies between the billing analysis and model results. To ensure that the site visits are effective, the pre-retrofit characteristics must be conducted on pre-retrofit sections of the participant building, or on future participant buildings prior to retrofit. DNV GL 11/10/16 Page 50

66 4.4 Low Income Multi Residential Affordable Housing () Low-Income Multi-Family Offering (Union) Program Type Relative Size Low Income 10% of utility shareholder incentive 7% of portfolio budget 3% of portfolio savings Offered Since 2012 Past Evaluations Savings are verified by third-party consultants annually on a sample of projects as part of the custom project savings verification (CPSV) process. On-site verification includes: 1) verifying installation date of equipment, 2) determining whether savings calculations were reasonable based on information available at the time of verification, 3) reviewing the assumptions used in the calculations, 4) discussing the difference between savings estimate and the verifier s estimate, and 5) recommending adjustments to the savings claim based on verification findings. A limited number of prescriptive measures have also been evaluated. *Percentages are of the combined portfolio, inclusive of both utilities Program description The programs offer multi-family low income housing customers with incentives to encourage energy efficient upgrades and funding for energy audits. The programs also provide technical services, benchmarking, and education for housing providers, building operators and tenants about their building s energy usage and ways to achieve energy efficiency. Eligible measures include boilers, ventilation systems, building envelope, window upgrades, in-suite water conservation measures (faucet aerators, showerheads), and heat reflector panels. The target markets for both programs are social and assisted housing providers who own and operate Part 3 16 buildings and private multi-residential building owners that provide housing to low income households. In addition, targets shelters and supportive housing Key objectives The overall objectives for the low income multi-family housing programs are to reduce energy costs for and provide non-energy benefits (e.g., improved indoor home comfort) for low income customers. Furthermore, energy efficiency improvements have the potential to contribute to housing affordability, preservation of the building stock, and the creation of healthier and more comfortable living environments for low income households. 16 Part 3 references buildings covered by Part 3 of the Ontario Building Code, defined as those exceeding 600 square meters in area or greater than three storeys in height; for residential energy efficiency programs, these are typically multifamily buildings. DNV GL 11/10/16 Page 51

67 The biggest challenge in this market is the matter of split incentives, whereby the building owner makes a significant financial investment in equipment or building upgrades, with limited or no ability to recoup their costs. Tenants, who are the ultimate consumers, do not have the same motivation to reduce their consumption. Other barriers that can impact the level of participation among building owners often include: Tenant/Landlord relationship; Access to capital or financing conditions; Uncertainty in market valuation of energy efficiency; Security and privacy concerns; Complexities and uncertainties in retrofit planning and implementation; Lack of confidence in retrofit performance and savings; Tolerance for adoption of new technologies; Health and safety issues; Lack of data on the building stock, or even for individual buildings; Social and multicultural diversity among residents; Building code violations or poor enforcement of building code; Pass through of beneficial improvements to tenants, e.g., rent adjustments; and, Municipal regulations (e.g., claw back of operating savings from energy efficiency by Service Manager on social housing units). In summary, the offering theory is as follows: In the short-term, the offering elements will lead social and assisted housing providers and low-income market rate multi-family housing providers to participate in the offering. The housing providers will learn about the offering and understand its benefits, and will request an energy audit and/or identify prescriptive and custom measures for implementation in their buildings. In the medium-term, prescriptive and custom measures will be installed in social and assisted housing and low-income market rate multi-family buildings. In the long-term, the offering will generate natural gas savings and customer satisfaction. The scorecard metrics affecting the shareholder incentive are shown in Table 4-7. Table scorecard metrics: Low Income Multi Residential Affordable Housing and Lowincome Multi-family Offering Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Union Gas Social and Assisted Multi- Family Cumulative Natural Gas Savings Market Rate Multi-Family Cumulative Savings 16.2 million CCM $914, million CCM $130,625 DNV GL 11/10/16 Page 52

68 Multi-Residential Cumulative Natural Gas Savings 64.9 million CCM $1,081,575 Source: EB / EB , Schedule B Program risks, impacts, and targeted evaluation activities There are several uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate and shareholder incentive amount. Table 4-8 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table 4-8. Risks, impacts, and evaluation activities: Low Income Multi Residential Affordable Housing and Low Income Multi-Family Offering Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Both Programs Jointly Pre-retrofit building characteristics, especially hidden features (e.g., wall insulation) are difficult to assess and may induce over- or underestimation of savings Medium Low Desk review of inputs and assumptions, Site visits Medium (site visits), Low (desk review) Measures might not be properly installed, which would affect performance Rebound effect and other behavioral changes not being captured in the netto-gross factor Low Low On-site inspections Medium Low Low Participant survey Low Free ridership assumptions may be off (0%, 1% to 5% Union)*, although this effect is typically very small for low income programs (but could be higher for market-rate rents housing) Savings assumptions of prescriptive measures Low Low Participant survey, Interviews with trade allies Medium Low Review of savings assumptions Low Low DNV GL 11/10/16 Page 53

69 Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Measures EUL assumptions being different from real useful life, affecting costeffectiveness and cumulative savings estimates Measures incremental cost assumptions being different from real incremental costs, affecting costeffectiveness results Medium Medium Persistence study, Review of EUL assumptions High High Market study, Review of cost assumptions High (persistence study), Low (review of assumptions) Medium (market study), Low (review of assumptions) Union Gas Only Baseline assumptions for equipment replacements Only Baseline assumptions for equipment replacements Low Medium Baseline study, Review of files, Participant survey (for early replacement vs ROB boilers) High High Baseline study, Review of files, Participant survey (for early replacement vs ROB boilers) Medium (baseline study), Low (review of files, participant survey) Medium (baseline study), Low (review of files, participant survey) *EB New and Updated DSM Measures - Joint Submission from Union Gas Ltd. and Gas Distribution (Exhibit B, Tab 1, Schedule 2, p. 3-4 of 19) Evaluation activity recommendation This section describes the potential evaluation activities in greater detail with a description of the study objectives and a high-level outline of the key tasks. Although the evaluation activities are outlined as discrete tasks, they can be combined to produce more comprehensive studies. DNV GL 11/10/16 Page 54

70 Traditionally, the low income multi-family programs have been included in the custom savings verification process to confirm savings estimates. The EC recommends that they continue to be included in that process in 2016, but that their inclusion be re-evaluated for It s possible that a desk review of files and participant surveys may be sufficient to confirm savings estimates. The EC also recommends initiating a market assessment to look at things such as pre-retrofit building characteristics and baseline boiler assumptions which will improve the building saving estimates. Finally, the EC also recommends a NTG evaluation to address free ridership and spillover in privately-owned low income multifamily buildings; however, this is a low priority and recommended for after the mid-term review. These activities (and other options) are described in greater detail in the following sections Participant survey and trade ally interviews Program free ridership and spillover are currently based on stipulated values. Although these values are usually very small for low income customers, these programs target building owners (private housing and social housing) that may present different market barrier levels and types, and free ridership levels, than their low income tenants. Finally, the baseline assumption for boilers may or may not accurately reflect the number of early replacements versus those that are replaced on burnout. The Boiler Baseline Study, currently in progress, may address this issue. If not, a participant survey can provide further insights on the appropriate baseline assumptions, in addition to measuring customer and trade ally satisfaction with the program offerings. This evaluation activity addresses these risks from Table 4-8: Free ridership assumptions: a well-crafted participant and trade ally survey can collect the data necessary to estimate program free ridership. Additional questions can be added to estimate spillover. Rebound effect and other behavioral changes: A survey can ask tenants to identify how they have changed behaviors and energy usage since the measures were installed. The survey would be conducted with a sample or tenants and a control group of non-participants to correct for exogenous trends. Boiler replacement assumptions: A survey can identify whether an installed boiler (or other equipment) was replace-on-burnout or early replacement. The objectives of participant survey would be to: Determine the free ridership rate Identify behavioral changes, including any rebound effect, that would have an impact on the energy savings estimate Determine the rate of boiler change-outs that are early replacement versus replace on burnout. Confirm customer and trade ally satisfaction with program offerings and identify potential improvements The participant survey should include the following tasks: Design a sample that represents the program population, oversampling boiler replacements Write and deliver a survey for tenants and building owners to address the research questions. Analyze the tenant survey data to determine satisfaction. Analyze the building owner survey data to determine free ridership, rate of early replacement, and satisfaction. Write a report on the methodology, results, and recommendations that may help improve program design. DNV GL 11/10/16 Page 55

71 Market studies Program measure life and equipment baseline are currently based on stipulated values. One or more market studies can gather data that more directly reflects the participants of the programs. This evaluation activity addresses these risks from Table 4-8: Incorrect measure life: A study that assesses effective useful life (EUL) using client surveys and site visits will quantify persistence and degradation of upgrades and savings. Alternatively, EULs can be validated at lower cost using secondary sources, such as other persistence studies or manufacturers data. Inaccurate incremental cost: A market study that identifies the incremental cost of an energy efficient installation relative to a standard efficiency installation in Ontario will result in more accurate cost-benefit study results. Equipment baseline assumptions: A market study that identifies the average installed equipment efficiency in Ontario will more accurately reflect the degree to which the program influences a change. The average installed efficiency may not be the same as the minimum efficiency available. The objectives of the market studies would be to: Improve the cumulative savings by more accurately determining measure life Identify the actual average boiler efficiency sold in the Ontario market to improve the building simulation results. The market study should include the following tasks: Investigate the potential data available for study: sales data, government data, program data. Write a work plan to clearly identify the boundaries of the study: what technologies will be studied, what portions of the market will be measured, what will the outcome be. Collect and analyze data according to the work plan. Write a report on the methodology, results, and recommendations that may help improve program design Other options The EC team has identified activities that could replace some of those outlined above for a different cost. This section outlines the other options and where they can be implemented. Desk review of files This evaluation activity addresses these risks from Table 4-8: Poorly assessed pre-retrofit building characteristics: Assumptions can be reviewed to determine whether they are reasonable. Equipment replacement baseline assumptions: Assumptions can be reviewed to determine whether they are reasonable, and if they follow program procedure where applicable. Savings assumptions of prescriptive measures: The savings assumptions of prescriptive measures would be reviewed to assess whether they are reasonable. The objective of the desk review would be to provide a third-party verification of the assumptions used in savings estimates in a cost-effective manner. DNV GL 11/10/16 Page 56

72 Site visits This evaluation activity addresses these risks from Table 4-8: Poorly assessed pre-retrofit building characteristics: Using technology such as infrared cameras, hidden pre-retrofit building characteristics might be more accurately measured. Improper installation: A site visit could inspect the work that was completed to ensure it was installed properly. The objective of the site visits would be to provide a third-party verification of the assumptions used in the savings estimates and confirm proper installation of the energy efficiency measures. To ensure that the site visits are effective, the pre-retrofit characteristics must be conducted on pre-retrofit sections of the participant building, or on future participant buildings prior to retrofit. 4.5 Low Income New Construction () Program Type Relative Size Offered Since Past Evaluations Low Income 1% of utility shareholder incentive 1% of portfolio budget < 1% of portfolio savings 2016 (2015 as a pilot) No evaluation activities (new program) *Percentages are of the combined portfolio, inclusive of both utilities Program description The offer is specifically directed to residential and multi-residential affordable building developments and efforts will focus on working with and through municipal governments and private and non-profit local housing corporations. It promotes the adoption of energy efficiency features and building practices among developers and builders of affordable housing projects in order to exceed Ontario Building Code (OBC) requirements by at least 15%. proposed to begin this offer as a pilot in 2015, rolling out for full scale deployment beginning in Key objectives The overarching goal of the Low Income New Construction offer is to promote the adoption of energy efficiency features and practices among developers and builders of affordable housing. This offer is designed to encourage municipalities to take a proactive role in incorporating energy efficiency standards in their own affordable housing plans and programs. The offer will provide financial incentives and enabling support to their affordable housing partners to facilitate the integration of energy efficiency in housing projects. Specific barriers to participation may include: 17 was unable to secure participants to undertake the pilot in 2015; nonetheless, the offer was approved for full-scale deployment in DNV GL 11/10/16 Page 57

73 The government grant funding for new construction under the federal/provincial Investment in Affordable Housing Program ( IAH ) poses financial challenges for builders to explore and/or incorporate energy efficiency and sustainability features that exceed building code requirements The expected building code change in 2017 will make it even more difficult to recognize savings above code While IAH Program Guidelines for Service Managers encourages enhanced energy efficiency features in project applications, there are neither pathways nor tools available in the IAH program to foster this outcome Private developers are likely to build residential units that will generate higher margins, and therefore, may be less willing to explore costlier energy efficient upgrades Applying for public funding subsidies can be cumbersome and onerous, which could discourage builders from participating in government-sponsored initiatives The scorecard metrics affecting the shareholder incentive are shown in Table 4-9. Table scorecard metrics: Low Income New Construction Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ New Construction Program Participants 6 buildings $240,350 Source: EB / EB , Schedule B. EB Exhibit I.T5.EGDI.LIEN.2 : Target participation was 3 single family homes and 2 multi-residential buildings. Since then the target has been increased to Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate (which is not a shareholder metric) and shareholder incentive amount. Table 4-10 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table Risks, impacts, and evaluation activities: Low Income New Construction Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost New program, no evaluation has been conducted yet High Medium Conduct first evaluation Medium Determination of baseline NC efficiency Medium Medium Baseline Study Medium DNV GL 11/10/16 Page 58

74 Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Proper installation of planned energy efficiency features Modeling accuracy, especially for very efficient homes and newer technologies High * Low On-site inspections Medium High Low Desk review Low Prescriptive path for Part 9 homes applied to varying house models Program participant count (for shareholder incentive) Free ridership assumptions may be off (current assumption unknown) Measure EUL assumptions being different from real useful life, affecting cost-effectiveness and cumulative savings estimates High Medium Desk review and modeling for homes following the prescriptive path Low Medium Verification of participant count, especially to ensure completion of project and no double-counting between periods Medium Medium Participant survey,** Interviews with trade allies Medium Medium Persistence study, Review of EUL assumptions Low Low Low High (persistence study), Low (review of assumptions) DNV GL 11/10/16 Page 59

75 Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Measure incremental cost assumptions being different from real incremental costs, affecting costeffectiveness results Medium Medium Market study, Review of cost assumptions Medium (market study), Low (review of assumptions) Part 9 references buildings covered by Part 9 of the Ontario Building Code, defined as those with three or fewer storeys and having a building area not exceeding 600 square meters; for residential energy efficiency programs, these are typically single family buildings. * Low for Part 9 because of Energy Star quality control. **There is currently not enough participation in this program to produce rigorous results with a survey Evaluation activity recommendation This section describes the potential evaluation activities in greater detail with a description of the study objectives and a high-level outline of the key tasks. Although the evaluation activities are outlined as discrete tasks, they can be combined to produce more comprehensive studies. For this program, the EC team recommends that a desk review be conducted when sufficient participation is achieved. We also recommend initiating a market assessment to look at the baseline for new housing units, incremental cost, and measure life; however, given the low participation in this program, the EC recommends that the market assessment happen after the mid-term review. These activities (and other options) are described in greater detail in the following sections. NOTE: This program only has a target participation of six buildings in Depending on real participation levels, evaluating impacts for this program might not be a high priority in the short term Desk review of files The desk review of participant files addresses these risks from Table 4-10: Prescriptive Path for Part 9 Homes: A review of savings would be conducted, for homes following the prescriptive path, using modeling. Modeling Accuracy: Modeling inputs and assumptions would be reviewed. Participant Count: Participant count would be reviewed, especially to ensure completion of projects and detect any double-counting, and verify that participants are meeting all program requirements. The objectives of this review would be to: Verify energy savings resulting from program participation. Verify the reasonableness of modeling inputs and screen for potential errors. Verify the participant count used to calculate shareholder incentives. The desk review of files would include the following tasks: The verification contractor will ask builders for blueprints and equipment specifications for a sample of developments built under the program and a listing of the measures that were installed to ensure the building exceeded building code. DNV GL 11/10/16 Page 60

76 The verification contractor will create two models of each building using a standard building simulation program: one that meets code, and one that reflects the building as it was built. The simulations will be run to estimate annual building energy use and the difference calculated to determine whether the building achieves the desired improvement metric On-site inspections On-site inspections for a sample of files would help verify proper installation of measures. For measures that are not accessible after completion of work (e.g., wall insulation), inspections could be conducted during construction (e.g., pre-gypsum inspection for wall insulation). Alternatively, thermography imaging could be used to detect deficiencies after a construction project is completed. This evaluation activity addresses this risk from Table 4-10: Improper installation: A site visit could inspect the work that was completed to ensure it was installed properly. The objective of on-site inspections would be to: Ensure energy efficiency features are procuring expected energy savings. The on-site inspections would include the following activities: Create a sample of construction projects. Inspect construction projects at varying construction stages. Write a report on the methodology, results, and recommendations that may help improve program design Participant survey and trade ally interviews Program free ridership and spillover are currently based on stipulated values. Although these values are usually very small for low income customers, these programs target developers and builders of affordable housing projects that may present different market barrier levels and types, and free ridership levels, than low income tenants. A participant survey can address these issues and investigate customer and trade ally satisfaction with the program offerings. This evaluation activity addresses these risks from Table 4-10: Free ridership assumptions: a well-crafted participant and trade ally survey can collect the data necessary to estimate program free ridership. Additional questions can be added to estimate spillover. The objectives of participant survey would be to: Determine the free ridership rate. The participant survey should include the following tasks: Write and deliver a survey to address the research questions. Analyze the survey data to determine free ridership and satisfaction. Write a report on the methodology, results, and recommendations that may help improve program design. Because of low anticipated participation levels, the EC team does not recommend that such a survey be conducted in the short term. A set of interviews oriented toward qualitative findings would be better suited to assist a process evaluation. DNV GL 11/10/16 Page 61

77 Market studies Program measure life and incremental costs are currently based on stipulated values. One or more market studies can gather data that more directly reflects the participants of the programs. This evaluation activity addresses these risks from Table 4-10: Determination of baseline NC efficiency: A baseline study would be conducted to assess the baseline efficiency of new construction in the affordable housing sector. Market baseline efficiency (the average efficiency of new buildings) can be different from minimal regulation efficiency. Incorrect measure life: A study that assesses effective useful life (EUL) using client surveys and site visits will quantify persistence and degradation of upgrades and savings. Alternatively, EULs can be validated at lower cost using secondary sources, such as other persistence studies. Inaccurate incremental cost: A market study that identifies the incremental cost of an energy efficient construction relative to a baseline construction in Ontario will result in more accurate costbenefit study results. The objectives of the market studies would be to: Improve the cumulative savings by more accurately determining measure life Improve the cost-benefit analysis by more accurately determining incremental cost. Identify the actual baseline for new construction in the Ontario market to improve the gross savings calculation. The market study should include the following tasks: Investigate the potential data available for study: sales data, government data, program data. Write a work plan to clearly identify the boundaries of the study: what technologies will be studied, what portions of the market will be measured, what will the outcome be. Collect and analyze data according to the work plan. Write a report on the methodology, results, and recommendations that may help improve program design. DNV GL 11/10/16 Page 62

78 4.7 Large Volume (Union) Program Type Relative Size Large Volume 5% of utility shareholder incentive ~4% of portfolio budget ~27% of portfolio savings Offered Since 2013 (previous iteration of the program was offered in 2012) Past Evaluations Gross savings are verified by third-party consultants annually on a sample of projects as part of the custom project savings verification (CPSV) process. On-site verification includes: 1) verifying installation date of equipment, 2) determining whether savings calculations were reasonable based on information available at the time of verification, 3) reviewing the assumptions used in the calculations, 4) discussing the difference between savings estimate and the verifier s estimate, and 5) recommending adjustments to the savings claim based on verification findings. *Percentages are of the combined portfolio, inclusive of both utilities Program description Union continues to encourage the adoption of energy efficient equipment, technologies, and actions through direct customer interaction via its Large Volume program. The Large Volume program is applicable to customers in Rate T2/Rate 100 only. The 2015 to 2020 program uses a direct access budget mechanism for the customer incentive budget process for Rate T2/Rate 100 customers. This mechanism grants each customer direct access to the customer incentive budget they pay in rates. Customers must use these funds to identify and implement energy efficiency projects, or lose the funds which will consequently become available for use by other customers in the same rate class. This use it or lose it approach ensures each customer has first access to the amount of incentive budget funded by their rates. The incentive approach for Rate T1 customers remains unchanged from the aggregate pool approach offered in The Large Volume program is the only direct access program offered in Ontario. It is similar in concept, though not funding mechanism, to the standard custom programs offered by the two gas utilities and to the electricity CDM Process and Systems program offered by electricity distributors. It also overlaps to some extent with the Custom Track of the electricity CDM Retrofit program Key objectives The goal of the Large Volume program is to generate long-term and cost-effective energy savings for Union s largest commercial and industrial customers. The large volume market is heterogeneous, with most projects tied directly to unique processes or technology requirements. Accordingly, most Large Volume projects are custom. The scorecard metrics affecting the shareholder incentive are shown in Table DNV GL 11/10/16 Page 63

79 Table scorecard metrics: Large Volume Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Union Gas Cumulative Natural Gas Savings 1.06 billion CCM $940,500 Source: EB / EB , Schedule B. Savings target estimated from 2015 spent budget, 2015 claimed savings, and 2016 proposed budget Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate and shareholder incentive amount. Table 4-12 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table Risks, impacts, and evaluation activities: Large Volume Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Union Gas Only Inaccurate savings estimates High High Third party verification High Inaccurate baselines Medium High Market assessment Medium to High Inaccurate operating assumptions Poor base case documentation High High Third party verification High Medium/High Third party verification High Medium to High Inaccurate free ridership assumptions High High Participant survey Low Inaccurate spillover assumptions High Low Participant survey with desk reviews Medium Inaccurate measure life assumptions Medium High Persistence study High Inaccurate incremental cost assumptions Low Medium Participant survey Market assessment Low (participant survey) Medium (market assessment) DNV GL 11/10/16 Page 64

80 Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Interconnection to prescriptive and quasiprescriptive measures Electricity and water savings interactions may not be captured Medium Medium Third party verification Knowledge and understanding of Prescriptive projects Participant survey Medium Low Third party verification Low High Evaluation activity recommendation This section describes the potential evaluation activities in greater detail with a description of the study objectives and a high-level outline of the key tasks. Although the evaluation activities are outlined as discrete tasks, they can be combined to produce more comprehensive studies. For example, the third party verification can be supplemented with participant surveys to provide a more streamlined evaluation that places fewer burdens on the customer. For this program, the EC recommends completing annual NTG and verification evaluations which will include telephone surveys and selective on-site visits for large or uncertain projects. The EC also recommends a cross-cutting persistence study across a number of C&I programs Savings verification Program gross savings are currently based on estimates developed by utility representatives in conjunction with trade allies and program participants. The Large Volume projects are termed custom projects because individual savings estimates are calculated for each project to reflect the specific equipment, operating conditions, and baseline conditions for that project. The program has traditionally used a thirdparty verification contractor, overseen by the utilities, to review the savings estimates and a third-party auditor to confirm the results of the verification contractor. Third party verification is the industry bestpractice for confirming the energy savings realized through custom energy efficiency programs; however, we the EC recommends a single contractor overseen by the OEB rather than a contractor and an auditor. This evaluation activity addresses these risks from Table 4-12: Inaccurate savings estimates. Changes in installation details between incentive approval (and tracking savings estimate calculation) and the evaluation are common. The third-party verification will identify the discrepancies and determine a more accurate estimate that reflects the actual installation. Inaccurate operating assumptions. Changes in operation between incentive approval (and tracking savings estimate calculation) and the evaluation are common. The third-party verification will identify the discrepancies and determine a more accurate estimate that reflects the actual equipment operation. Poor base case documentation. A predominance of larger and process-related projects suggests a higher likelihood of uncertainty related to baselines. Program and project documentation is rarely organized to facilitate evaluation; it is typically configured to comply with program administration and incentive procedures. In cases where the base case documentation is not fully informative, the DNV GL 11/10/16 Page 65

81 evaluation will incur additional uncertainties. The third-party verification can be used to fill in the blanks and ensure a more accurate savings estimate. Interconnection to C&I behavior programs. The services offered by C&I behavior or market transformation programs like Run Smart and Strategic Energy Management may interact with the energy savings from the projects that receive incentives under the Large Volume program. Interconnection to prescriptive and quasi-prescriptive measures. Projects, as defined by the customer, may include prescriptive and custom measures from a program perspective. The verification study will need to be clear about what portion of the project each question is meant to address, and interactive effects between custom and prescriptive savings should be taken into account. In addition, the presence of custom and prescriptive measures at one site should cause the verification contractors to coordinate to minimize burden on the customer. Electricity and water savings interactions may not be captured. The installation of natural gas energy efficiency measures (such as low-flow showerheads in employee locker rooms) can result in electric or water savings which must be taken into account in the cost effectiveness calculation but may not be captured in program tracking or documentation. The third-party verification can be used to fill in the blanks and ensure a more accurate representation of program benefits. The objectives of the third party verification would be to: Verify that the energy efficiency measures were installed as reported and continue to operate as designed Verify the energy savings resulting from the installed energy efficiency measures Identify systematic program improvements that can increase the accuracy of program-produced energy savings estimates going forward. The third party verification should include the following tasks: Request and receive tracking data files and project documentation sufficient to identify the size and type of project installed as well as any services received through or projects promoted by the Run Smart, Strategic Energy Management, or C&I Prescriptive programs. Design a sophisticated sample and ensure that it represents the program and achieves a precision of 90/10. Conduct telephone calls, desk reviews, and on-site visits to verify installation and confirm the gross energy savings. Produce site-level reports that describe the measure, the savings calculation, the assumptions used in the calculation, the information found during the verification, and any changes to the energy savings estimate. Write a report that summarizes the results of the study and provides recommendations for improvement Participant surveys Program net savings are currently based on a stipulated NTG factor. A participant survey can use industrystandard data collection methods to identify the influence the program had on the participant s decision to install an energy efficient measure for a sample of program participants. This evaluation activity addresses these risks from Table 4-12: Inaccurate free ridership assumptions. Inaccurate free ridership assumptions will result in an incomplete understanding of the influence and value of the energy efficiency program. The participant DNV GL 11/10/16 Page 66

82 survey will produce an accurate free ridership estimate specific to Ontario and the evaluated program year. Inaccurate spillover assumptions. Inaccurate spillover assumptions will result in an incomplete understanding of the influence and value of the energy efficiency program. The participant survey will produce an accurate spillover estimate specific to Ontario and the evaluated program year. Inaccurate incremental measure cost assumptions. A participant survey can collect accurate information on the incremental cost related to custom projects, which will result in more accurate costbenefit study results. Interconnection to C&I behavior programs. The services offered by C&I behavior or market transformation programs like Run Smart and Strategic Energy Management may influence the customer s decision to install projects under the Large Volume program. A participant survey can capture that influence and accurately represent it in the NTG adjustment. Interconnection to prescriptive and quasi-prescriptive measures. Projects, as defined by the customer, may include prescriptive and custom measures from a program perspective. A participant survey can separate the two projects in the customer s mind and accurately identify the NTG adjustment specific to the Large Volume program. The objectives of the participant survey would be to: Determine the free ridership and spillover associated with the installation of the energy efficiency measures. Gather information on incremental measure cost. Distinguish the effects of the Large Volume program from the Run Smart, Strategic Energy Management, and C&I Prescriptive programs. Understand the influence that participation in the Run Smart, Strategic Energy Management, and C&I Prescriptive programs had on the customer s decision to install the energy efficiency measure. The participant survey should include the following tasks: Request and receive tracking data files and project documentation sufficient to identify the size and type of project installed as well as any services received through or projects promoted by the Run Smart, Strategic Energy Management, and C&I Prescriptive programs. Design a sophisticated sample and ensure that it represents the program and achieves a precision of 90/10. Write a telephone survey and deliver it to project decision-makers to determine the effect of the Large Volume program on the company s decision to install an energy efficiency measure; particularly the effect on the timing, efficiency, and size of the measure. - Notify the interviewer of the services received through or projects promoted by the Run Smart, Strategic Energy Management, and C&I Prescriptive programs so the interviewer can properly account for them during the customer data collection. - Probe for the influence of the direct access funding mechanism and long-term utility relationships on the decision to install. - Ask about internal policies on simple payback or rate of return for approving energy and non-energy projects. - Probe for non-energy benefits sold by the program when measuring influence. DNV GL 11/10/16 Page 67

83 Identify measures that were installed without an incentive, but that could be as a result of the company s program participation (spillover). If spillover exists, conduct follow-up phone calls by engineers to gather the data necessary to estimate the energy savings resulting from the measure installed. Conduct a ratio estimation analysis to expand the sample results to the population. Write a comprehensive report on the overall program free ridership and spillover, assembled into a representative NTG rate. Examine the free ridership and spillover to provide information that may help improve program design. Make recommendations to improve the quality of the program s energy savings calculations in future program years Market assessment Baseline assumptions for custom project energy savings estimates are currently determined on a case-bycase basis, but it is unclear what informs those selections. Market assessments can be done on a broad level for popular measures such as boilers or a case-by-case level for unique projects. This evaluation activity addresses these risks from Table 4-12: Inaccurate baselines. The baseline for custom projects may not be immediately identifiable, especially for innovative or process-related energy efficiency projects. A market assessment, whether done for a commonly installed project or on a case-by-case basis, can produce a more accurate understanding of standard practice and the appropriate calculation baseline. Inaccurate incremental measure cost assumptions. A market assessment can produce the average or typical incremental cost for commonly installed custom measures, which will result in more accurate cost-benefit study results. Using a market assessment for common measures could result in a lower cost and higher accuracy than a project-by-project study using participant surveys. The objectives of the market assessments would be to: Improve the cumulative savings by more accurately determining the measure baseline. Improve the cost-benefit analysis by more accurately determining incremental cost. The baseline portion of the market study should include the following tasks: For generalized baseline assumptions for a single measure across a number of installations: - Investigate the potential data available for study, such as sales data, government data, or program data. - Write a work plan to clearly identify the boundaries of the study: what technologies will be studied, what portions of the market will be measured, what will the outcome be. - Collect and analyze data according to the work plan. - Write a report on the methodology, results, and recommendations that may help improve program design. For a custom, single-measure baseline assumption at a single location (or for a single customer): - Identify market actors familiar with the process or equipment affected. - Conduct surveys with one or more market actors to collect data on the typical or standard practice installation under similar conditions. - Use the baseline to produce a more accurate energy savings estimate as part of the third-party verification. DNV GL 11/10/16 Page 68

84 - Document the process, data collected, and result in the site-level report. The incremental cost portion of the market study should include the following tasks: Investigate the potential data available for study, such as sales data, government data, or program data. Write a work plan to clearly identify the boundaries of the study, such as what technologies will be studied, what portions of the market will be measured, what will the outcome be. Collect and analyze data according to the work plan. Write a report on the methodology, results, and recommendations that may help improve program design Persistence study Custom measure lives are determined using either the filed EUL table (as a guide) or customer-specific information, which must be documented with appropriate references and/or other forms of substantiation. In general, assumptions in an EUL table are based on the mechanical life of the equipment installed (or replaced) but are not based on the actual operational life of the equipment. A persistence study can verify both assumptions and produce an accurate measure life assumption specific to Ontario. This evaluation activity addresses this risk from Table 4-12: Inaccurate measure life assumptions. A persistence study can identify how changes over the mechanical life of an installed measure can affect the energy savings realized by that measure, such as changes in operation practices, changes in building occupancy or tenant, and efficiency degradation. Section 5.4 contains additional information on persistence studies and how they are conducted Knowledge and understanding of other programs projects Union and offer a number of programs directed to the C&I sector in addition to Large Volume, such as the C&I prescriptive program, Strategic Energy Performance, and Run Smart. Although the utilities (and OEB and EC) generally treat the programs as distinct, customer participation in one program may influence their decision to participate in other programs, and customers may not be able to distinguish services from one program from services from another program. Knowing which customers participate in other programs and understanding the services they received and measures they installed provides a more complete picture which aids in data collection and spillover and free ridership estimates. This evaluation activity addresses these risks from Table 4-12: Interconnection to C&I behavior programs. The services offered by C&I behavior or market transformation programs like Run Smart and Strategic Energy Management may influence the customer s decision to install projects and interact with the energy savings under the Large Volume program. Interconnection to prescriptive and quasi-prescriptive measures. The services offered by the C&I Prescriptive program may influence the customer s decision to install projects and interact with the energy savings under the Large Volume program. The objectives of knowing and understanding other programs projects would be to: Provide the best available information on customer participation to aid and improve accurate data collection. Identify how the various C&I programs interact and the effect of those interactions. Facilitate accurate Large Volume verification and NTG results. DNV GL 11/10/16 Page 69

85 Knowing and understanding other programs projects should include the following tasks: Request and receive tracking data files and project documentation sufficient to identify the services received through or projects promoted by the Run Smart, Strategic Energy Management, and C&I Prescriptive programs. Cross-reference the data to identify customers who participated in more than one program and identify how they participated and what services they received. Summarize the data for data collection and third party verification. Add a section to the third party verification and NTG reports that identify any interactions and their effects. 4.8 Commercial and Industrial Prescriptive Offering () Prescriptive and Quasi-Prescriptive Offering (Union) Program Type Relative Size Resource Acquisition 9% of utility shareholder incentive ~10% of portfolio budget 14% of portfolio savings Offered Since 2011 Past Evaluations No evaluation activities to date *Percentages are of the combined portfolio, inclusive of both utilities Program description The Commercial/Industrial Prescriptive offering provides customers with a list of recommended technologies that have pre-determined incentive and savings amounts, defined by facility type and equipment size. The application process for the offering promotes ease of participation as no additional analysis or savings calculations are required. This also allows commercial, institutional, and small industrial customers with multiple facilities the option of rolling out technologies to an entire portfolio of buildings in an efficient way. The Commercial Prescriptive offer, administered within its Resource Acquisition (RA) program, is designed to capture energy savings through the installation of energy efficient technologies that have predetermined energy savings values. Opportunities for commercial customers include incentives to invest in energy efficient technologies in commercial buildings, such as the purchase and installation of efficient heating, ventilation, and air conditioning (HVAC) systems. Previously, customers had the opportunity to receive incentives for energy audits and energy management offers that focused on operational improvements to support savings opportunities. Union offers DSM prescriptive and quasi-prescriptive measures to more than 120,000 C&I customers. These customers are made up of office, retail, multi-family residential, food service, hotel/motel, manufacturing, agriculture, warehouse, entertainment & recreation, and education & health care segments. The prescriptive measures have pre-determined deemed savings based on the size and classification of the equipment while the quasi-prescriptive measures have one or more variable inputs that need to be known for each DNV GL 11/10/16 Page 70

86 installation in order to determine natural gas savings. An example of an input is the size or rating of the equipment (e.g., CFM or BTU) Key objectives The overall objectives for the Commercial/Industrial Prescriptive offering are to maximize cost-effective natural gas savings for all C&I customers while achieving long-term energy savings in commercial, institutional and industrial facilities. The scorecard metrics affecting the shareholder incentive are shown in Table Table scorecard metrics: Commercial and Industrial Prescriptive Offerings Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Union Gas Cumulative Natural Gas Savings Cumulative Natural Gas Savings 1,214.1 million CCM/year (all resource acquisition programs) million CCM/year (C&I Prescriptive 2016 target) million CCM (all large volume customers) million CCM (all small volume customers) million CCM (C&I Prescriptive 2016 target) $1,189,429 $738,482 Source: Schedule B Decision and order OEB-approved scorecard weighting Gas Distribution, Inc. (EB ) / Union Gas Limited (EB ) January 20, Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate and shareholder incentive amount. Table 4-14 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. DNV GL 11/10/16 Page 71

87 Table Risks, impacts, and evaluation activities: Commercial and Industrial Prescriptive Offerings Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost TRM savings do not reflect actual mix of program participation Unconfirmed measure installation Inaccurate application of TRM savings Medium Medium Participant survey Third party verification Low High Third party verification Medium Low to Medium Tracking data review and certification Low (participant survey) Medium (third party verification) Medium Low Inaccurate free ridership assumptions High High Participant survey Low Inaccurate spillover assumptions High Medium Participant survey with desk reviews Medium Inaccurate measure life assumptions Medium High Persistence study High Inaccurate incremental cost assumptions Interconnection to C&I behavior programs Low Medium Participant survey Market assessment Medium Medium Third party verification Knowledge and understanding of C&I behavior projects Participant survey Low (participant survey) Medium (market assessment) Low DNV GL 11/10/16 Page 72

88 Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Interconnection to C&I Custom and Large Volume measures Medium Medium Third party verification Knowledge and understanding of Custom and Large Volume projects Participant survey Low Electricity and water savings interactions may not be captured Medium Low Third party verification High Evaluation activity recommendation This section describes the potential evaluation activities in greater detail with a description of the study objectives and a high-level outline of the key tasks. Although the evaluation activities are outlined as discrete tasks, they can be combined to produce more comprehensive studies. For example, a participant survey could be combined with third party verification to estimate net and gross savings together. For these programs, the EC recommends that an annual verification study be conducted to confirm measure installation and operation. The EC recommends telephone verification supplemented with on-site verification for large or unusual projects. The verification study should include tracking data review and certification to ensure that the TRM estimates are being applied correctly. In 2017, we recommend a NTG study of 2016 program participants Tracking data review and certification Program-level gross savings are equal to the sum of individual measure installations with energy savings estimates based on the number of units installed and the energy savings prescribed by the TRM. In the past, a sample of program measures was reviewed by a third party to confirm that the TRM savings and adjustment factors were applied correctly. The EC recommends that this activity be conducted on a census of program measures. This evaluation activity addresses this risk from Table 4-14: Inaccurate application of TRM savings. Errors may occur when tracking energy efficiency measures and applying prescribed, per-unit TRM estimates to determine the measure-level and program-level savings estimate. A tracking data review and certification can identify those errors and help correct them. The objectives of the tracking data review and certification would be to: Identify errors in the tracking data that lead to an inaccurate program-level savings estimate. Work with the program to correct the errors and improve data tracking practices going forward. The tracking data review and certification should include the following tasks: DNV GL 11/10/16 Page 73

89 Request and receive tracking data files for all of the measures installed through the C&I prescriptive program in a given program year. Request and receive the scorecard savings results for the same program years, as well as instructions on how to identify which measures fall into which program period. Review the population of entries in the tracking data to identify the measure installed, quantity installed, and per-unit savings estimate used to determine the measure-level savings. Compare the per-unit savings estimates to the TRM to ensure proper application. Recreate the program-level savings calculation to verify the program-produced value. Communicate errors to the program and provide recommendations on how to avoid them in the future. Add a report section to the third party verification report to summarize the results of the certification Savings verification Program gross savings are currently based on prescribed or deemed savings from the TRM and an assumption that the measures were installed as intended. A savings verification effort can confirm the installation and collect data that can improve the measure-level savings estimates. This evaluation activity addresses these risks from Table 4-14: TRM savings do not reflect actual mix of program participation. TRM savings are generally based on secondary source data collected in other jurisdictions and based on a typical installation which may or may not represent the customers participating in the program. Third party verification can gather data to confirm or refute TRM assumptions. Unconfirmed measure installation. C&I projects do not always get installed in the quantity and applications in which they were originally intended. Changes in operation, processes, and design can affect the final energy efficiency measure, or unsatisfactory measures may be removed. Third party verification can confirm continued installation and operation. Interconnection to C&I behavior programs. The services offered by C&I behavior or market transformation programs like Run it Right, Run Smart, Comprehensive Energy Management, and Strategic Energy Management may interact with the energy savings from the projects that receive incentives under the C&I Prescriptive program. Interconnection to Custom and Large Volume measures. Projects, as defined by the customer, may include prescriptive and custom measures from a program perspective. The verification study will need to be clear about what portion of the project each question is meant to address, and interactive effects between custom and prescriptive savings should be taken into account. In addition, the presence of custom and prescriptive measures at one site should cause the verification contractors to coordinate to minimize burden on the customer. Electricity and water savings interactions may not be captured. The installation of natural gas energy efficiency measures (such as low-flow showerheads) can result in electric or water savings which must be taken into account in the cost effectiveness calculation but may not be captured in program tracking or documentation. The third-party verification can be used to fill in the blanks and ensure a more accurate representation of program benefits. The objectives of the third party verification would be to: Verify that the energy efficiency measures were installed as reported and continue to operate as designed. Collect data to confirm or refute the assumptions used for energy savings in the TRM. DNV GL 11/10/16 Page 74

90 The third party verification should include the following tasks: Request and receive tracking data files and project documentation sufficient to identify the size and type of project installed as well as any services received through or projects promoted by the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, or C&I Custom programs. Design a sophisticated sample and ensure that it represents the program and achieves a precision of 90/10. Conduct telephone calls, desk reviews, and on-site visits to verify installation and gather data representing typical operation Write a report that summarizes the results of the study Participant surveys Program net savings are currently based on a stipulated NTG factor. A participant survey can use industrystandard data collection methods to identify the influence the program had on the participant s decision to install an energy efficient measure for a sample of program participants. Free ridership and spillover are likely to be measure-specific, based on the technology, market conditions and specific program offering at specific times. Participant surveys can also collect other data to increase the accuracy of the energy savings result. This evaluation activity addresses these risks from Table 4-14: TRM savings do not reflect actual mix of program participation. TRM savings are generally based on secondary source data collected in other jurisdictions and based on a typical installation which may or may not represent the customers participating in the program. A participant survey can gather data to confirm or refute TRM assumptions. Inaccurate free ridership assumptions. Inaccurate free ridership assumptions will result in an incomplete understanding of the influence and value of the energy efficiency program. The participant survey will produce an accurate free ridership estimate specific to Ontario and the evaluated program year. Inaccurate spillover assumptions. Inaccurate spillover assumptions will result in an incomplete understanding of the influence and value of the energy efficiency program. The participant survey will produce an accurate spillover estimate specific to Ontario and the evaluated program year. Inaccurate incremental measure cost assumptions. A participant survey can collect information on the incremental cost related to custom projects, which will result in more accurate cost-benefit study results. Interconnection to C&I behavior programs. The services offered by C&I behavior or market transformation programs like Run it Right, Run Smart, Comprehensive Energy Management, and Strategic Energy Management influence the customer s decision to install projects under the C&I Prescriptive program. A participant survey can capture that influence and accurately represent it in the NTG adjustment. Interconnection to Custom and Large Volume measures. Projects, as defined by the customer, may include prescriptive and custom measures from a program perspective. A participant survey can separate the two projects in the customer s mind and accurately identify the NTG adjustment specific to the C&I Prescriptive program. The objectives of the participant survey would be to: DNV GL 11/10/16 Page 75

91 Determine the free ridership and spillover associated with the installation of the energy efficiency measures. Gather information on incremental measure cost if available. Distinguish the effects of the C&I Prescriptive program from the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, and C&I Custom programs. Understand the influence that participation in the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, and C&I Custom programs had on the customer s decision to install the energy efficiency measure. Collect data to confirm or refute the assumptions used for energy savings in the TRM. The participant survey should include the following tasks: Request and receive tracking data files and project documentation sufficient to identify the size and type of project installed as well as any services received through or projects promoted by the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, and C&I Custom programs. Design a sophisticated sample and ensure that it represents the program and achieves a precision of 90/10. Write a telephone survey and deliver it to project decision-makers to determine the effect of the program on the company s decision to install an energy efficiency measure; particularly the effect on the timing, efficiency, and size of the measure. - Notify the interviewer of the services received through or projects promoted by the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, and C&I Custom programs so the interviewer can properly account for them during the customer data collection. - Ask about internal policies on simple payback or rate of return for approving energy and non-energy projects. - Probe for non-energy benefits sold by the program when measuring influence. Identify measures that were installed without an incentive, but that could be as a result of the company s program participation (spillover). If spillover exists, conduct follow-up phone calls by engineers to gather the data necessary to estimate the energy savings resulting from the measure installed. Conduct a ratio estimation analysis to expand the sample results to the population. Write a comprehensive report on the overall program free ridership and spillover, assembled into a representative NTG rate. Examine the free ridership and spillover to provide information that may help improve program design. Make recommendations to improve the quality of the program s energy savings calculations in future program years Market assessment Incremental costs for the cost-benefit analysis are currently based on stipulated values. A market assessment can gather data that more directly reflects the participants of the Ontario C&I Prescriptive program. This evaluation activity addresses this risk from Table 4-14: Inaccurate incremental measure cost assumptions. A market assessment can produce the average or typical incremental cost for commonly installed custom measures, which will result in more accurate cost-benefit study results. Using a market assessment for common measures could result in a lower cost and higher accuracy than a project-by-project study using participant surveys. DNV GL 11/10/16 Page 76

92 The objective of the market assessment would be to: Improve the cost-benefit analysis by more accurately determining incremental cost. The market assessment should include the following tasks: Investigate the potential data available for study, such as sales data, government data, or program data. Write a work plan to clearly identify the boundaries of the study, such as what technologies will be studied, what portions of the market will be measured, what will the outcome be. Collect and analyze data according to the work plan. Write a report on the methodology, results, and recommendations that may help improve program design Persistence study It is unclear how the measure life assumptions for prescriptive projects are determined. In general, such assumptions are based on the mechanical life of the equipment installed (or replaced) but are not based on the actual operational life of the equipment. A persistence study can verify both assumptions and produce an accurate measure life assumption specific to Ontario. This evaluation activity addresses this risk from Table 4-14: Inaccurate measure life assumptions. A persistence study can identify how changes over the mechanical life of an installed measure can affect the energy savings realized by that measure, such as changes in operation practices, changes in building occupancy or tenant, and efficiency degradation. Section 5.4 contains additional information on persistence studies and how they are conducted Knowledge and understanding of other programs projects Union and offer a number of programs directed to the C&I sector in addition to C&I Prescriptive, such as the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, and C&I Custom programs. Although the utilities (and OEB and EC) generally treat the programs as distinct, customer participation in one program may influence their decision to participate in other programs, and customers may not be able to distinguish services from one program from services from another program. Knowing which customers participate in other programs and understanding the services they received and measures they installed provides a more complete picture which aids in data collection and spillover and free ridership estimates. This evaluation activity addresses these risks from Table 4-14: Interconnection to C&I behavior programs. The services offered by C&I behavior or market transformation programs like Run it Right, Run Smart, Comprehensive Energy Management, and Strategic Energy Management may influence the customer s decision to install projects and interact with the energy savings under the C&I Prescriptive program. Interconnection to C&I Custom and Large Volume measures. The services offered by the C&I Custom or Large Volume programs may influence the customer s decision to install projects and interact with the energy savings under the C&I Prescriptive program. The objectives of knowing and understanding other programs projects would be to: Provide the best available information on customer participation to aid and improve accurate data collection. Identify how the various C&I programs interact and the effect of those interactions. Facilitate accurate C&I Prescriptive verification and NTG results. DNV GL 11/10/16 Page 77

93 Knowing and understanding other programs projects should include the following tasks: Request and receive tracking data files and project documentation sufficient to identify the services received through or projects promoted by the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, Large Volume, and C&I Custom programs. Cross-reference the data to identify customers who participated in more than one program and identify how they participated and what services they received. Summarize the data for data collection and third party verification. Add a section to the third party verification and NTG reports that identify any interactions and their effects. 4.9 Commercial and Industrial Direct Install Programs ( and Union) Program Type Relative Size Resource Acquisition 2% of utility shareholder incentive ~7% of portfolio budget ~2% of portfolio savings Offered Since 2016 Past Evaluations No evaluation activities to date (new program) *Percentages are of the combined portfolio, inclusive of both utilities Program description The Direct Install (DI) program offered by both utilities provides commercial and industrial customers with financial incentives for a set list of natural gas reducing technologies and equipment. The direct install programs differ from the prescriptive programs in that the gas utilities involvement will go one step further and provide customers with turnkey installation. The gas utilities direct install programs provide commercial and industrial customers with a one-stop shopping option where the customer can receive both the financial incentives and installation assistance required to seamlessly upgrade current equipment and technologies to more efficient options Key objectives The key objective of the Commercial and Industrial DI Programs is to achieve cost-effective savings at customer facilities that might otherwise not participate in a prescriptive or custom program. Straightforward measures are deployed with a general goal of maximizing volume of savings relative to deployment effort. The scorecard metrics affecting the shareholder incentive are shown in Table DNV GL 11/10/16 Page 78

94 Table scorecard metrics: Commercial and Industrial Direct Install Programs Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Union Gas Cumulative Natural Gas Savings Cumulative Natural Gas Savings 1,214.1 million CCM/year (all resource acquisition programs) 6.7 million CCM/year (C&I DI 2016 target) million CCM (all large volume customers) million CCM (all small volume customers) 66.4 million CCM (C&I DI 2016 target) $26,380 $334,028 Source: Schedule B Decision and order OEB-approved scorecard weighting Gas Distribution, Inc. (EB ) / Union Gas Limited (EB ) January 20, Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate and shareholder incentive amount. Table 4-16 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table Risks, impacts, and evaluation activities: Commercial and Industrial Direct Install Programs Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost TRM savings do not reflect actual mix of program participation Inaccurate measure life assumptions Medium Medium Participant survey Third party verification Medium High Third party verification Persistence study Low (participant survey) Low (third party verification) Low (third party verification) High (persistence study) DNV GL 11/10/16 Page 79

95 Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Inaccurate application of TRM savings High Low to Medium Tracking data review and certification Low Inaccurate free ridership assumptions Low Medium Participant survey Medium Inaccurate spillover assumptions Low Low Participant survey with desk reviews Medium Electricity and water savings interactions may not be captured Medium Low Third party verification Medium Evaluation activity recommendation This section describes the potential evaluation activities in greater detail with a description of the study objectives and a high-level outline of the key tasks. Although the evaluation activities are outlined as discrete tasks, they can be combined to produce more comprehensive studies. For this program, the EC recommends completing an evaluation in 2017, or once energy savings are realized. The evaluation should include tracking data review and certification, third party verification, and a participant study to determine a program-specific NTG adjustment factor Tracking data review and certification Program-level gross savings are equal to the sum of individual measure installations with energy savings estimates based on the number of units installed and the energy savings prescribed by the TRM. The program-level savings are not currently certified by a third party to confirm that the tracking data correctly applies the TRM estimates and produces the program-level sum. This evaluation activity addresses this risk from Table 4-16: Inaccurate application of TRM savings. Errors may occur when tracking energy efficiency measures and applying prescribed, per-unit TRM estimates to determine the measure-level and program-level savings estimate. A tracking data review and certification can identify those errors and help correct them. The objectives of the tracking data review and certification would be to: Identify errors in the tracking data that lead to an inaccurate program-level savings estimate. Work with the program to correct the errors and improve data tracking practices going forward. The tracking data review and certification should include the following tasks: DNV GL 11/10/16 Page 80

96 Request and receive tracking data files for all of the measures installed through the C&I DI program in a given program year. Request and receive the scorecard savings results for the same program years, as well as instructions on how to identify which measures fall into which program period. Review the population of entries in the tracking data to identify the measure installed, quantity installed, and per-unit savings estimate used to determine the measure-level savings. Compare the per-unit savings estimates to the TRM to ensure proper application. Recreate the program-level savings calculation to verify the program-produced value. Communicate errors to the program and provide recommendations on how to avoid them in the future. Add a report section to the third party verification report to summarize the results of the certification Savings verification Program gross savings are currently based on prescribed or deemed savings from the TRM and an assumption that the measures were installed (and continue to be installed) as intended. A savings verification effort can confirm continued installation and collect data that can improve the measure-level savings estimates. This evaluation activity addresses these risks from Table 4-16: TRM savings do not reflect actual mix of program participation. TRM savings are generally based on secondary source data collected in other jurisdictions and based on a typical installation which may or may not represent the customers participating in the program. Third party verification can gather data to confirm or refute TRM assumptions. Inaccurate measure life assumptions. Directly-installed measures may be removed by the building owner/occupant after being installed by the program. Third party verification can confirm continued installation and operation. Electricity and water savings interactions may not be captured. The installation of natural gas energy efficiency measures (such as low-flow showerheads) can result in electric or water savings which must be taken into account in the cost effectiveness calculation but may not be captured in program tracking or documentation. The third-party verification can be used to fill in the blanks and ensure a more accurate representation of program benefits. The objectives of the third party verification would be to: Verify that the energy efficiency measures continue to be installed as reported and operate as designed. Collect data to confirm or refute the assumptions used for energy savings in the TRM. The third party verification should include the following tasks: Request and receive tracking data files and project documentation sufficient to identify the size and type of project installed. Design a sample and ensure that it represents the program and achieves a precision of 90/10. Conduct telephone calls and desk reviews to verify installation and gather data representing typical operation Write a report that summarizes the results of the study Participant surveys Program net savings are currently based on a stipulated NTG factor. A participant survey can use industrystandard data collection methods to identify the influence the program had on the participant s decision to install an energy efficient measure for a sample of program participants. Free ridership and spillover are DNV GL 11/10/16 Page 81

97 likely to be measure-specific, based on the technology, market conditions and specific program offering at specific times. Participant surveys can also collect other data to increase the accuracy of the energy savings result. This evaluation activity addresses these risks from Table 4-16: TRM savings do not reflect actual mix of program participation. TRM savings are generally based on secondary source data collected in other jurisdictions and based on a typical installation which may or may not represent the customers participating in the program. A participant survey can gather data to confirm or refute TRM assumptions. Inaccurate free ridership assumptions. Inaccurate free ridership assumptions will result in an incomplete understanding of the influence and value of the energy efficiency program. The participant survey will produce an accurate free ridership estimate specific to Ontario and the evaluated program year. Inaccurate spillover assumptions. Inaccurate spillover assumptions will result in an incomplete understanding of the influence and value of the energy efficiency program. The participant survey will produce an accurate spillover estimate specific to Ontario and the evaluated program year. The objectives of the participant survey would be to: Determine the free ridership and spillover associated with the energy efficiency measures. Collect data to confirm or refute the assumptions used for energy savings in the TRM. The participant survey should include the following tasks: Request and receive tracking data files and project documentation sufficient to identify the size and type of project installed. Design a sample and ensure that it represents the program and achieves a precision of 90/10. Write a telephone survey and deliver it to site contacts to confirm continued measure installation and determine whether they would have installed the measure on their own. Identify measures that were installed outside of the program and without an incentive, but that could be as a result of the company s program participation (spillover). If spillover exists, conduct follow-up phone calls by engineers to gather the data necessary to estimate the energy savings resulting from the measure installed. Conduct a ratio estimation analysis to expand the sample results to the population. Write a comprehensive report on the overall program free ridership and spillover, assembled into a representative NTG rate. Examine the free ridership and spillover to provide information that may help improve program design. Make recommendations to improve the quality of the program s energy savings calculations in future program years Persistence study The measure lives used for the C&I DI program are currently based on prescribed or deemed savings from the TRM and an assumption that the measures will remain installed as intended. A persistence study can produce an accurate measure life assumption. This evaluation activity addresses this risk from Table 4-16: Inaccurate measure life assumptions. A persistence study can identify how changes over the mechanical life of an installed measure can affect the energy savings realized by that measure, such as changes in operation practices, changes in building occupancy or tenant, and efficiency degradation. Section 5.4 contains additional information on persistence studies and how they are conducted. DNV GL 11/10/16 Page 82

98 4.10 Commercial and Industrial Custom Program ( & Union) Program Type Relative Size Resource Acquisition 31% of utility shareholder incentive ~17% of portfolio budget 44% of portfolio savings Offered Since 2011 Past Evaluations Gross savings are verified by third-party consultants annually on a sample of projects as part of the Custom project savings verification (CPSV) process. On-site verification includes: 1) verifying installation date of equipment, 2) determining whether savings calculations were reasonable based on information available at the time of verification, 3) reviewing the assumptions used in the calculations, 4) discussing the difference between savings estimate and the verifier s estimate, and 5) recommending adjustments to the savings claim based on verification findings. *Percentages are of the combined portfolio, inclusive of both utilities Program description The custom program offerings have been designed to encourage commercial and industrial customers to reduce their energy consumption by providing customer-specific energy efficiency and conservation solutions. The custom programs provide financial incentives, technical expertise, and guidance with respect to energy related decision making and business justification, including helping customers to prioritize energy efficiency projects against their own internal competing factors and demonstrate the competitive advantage customers can gain through efficiency upgrades. These custom programs differ from the prescriptive and direct install programs as they provide tailored services and varying financial incentives based on overall natural gas savings realized by the customer to address customer-specific needs. The custom programs build upon those deployed by the gas utilities in past. They are very similar to, and serve effectively the same customers as, the electricity CDM Retrofit Program s Custom Track. The goal of the Commercial Custom offer is to reduce natural gas use through the capture of energy efficiency opportunities in commercial buildings, including retrofits of building components and upgrades at the time of replacement. The offer aims to promote the highest level of energy efficiency. The Industrial Custom Solutions offer is designed to capture cost-effective energy savings within the industrial sector by delivering customized energy solutions aimed at supporting customers through a continuous improvement approach. Industrial Energy Solutions Consultants (ESCs) focus on assisting customers with the adoption of energy efficient technologies by overcoming financial, knowledge or technical barriers. Union focuses on advancing customer energy efficiency and productivity by providing a mix of custom incentives, education and awareness to C&I customers across all segments. The objective of the Custom offering is to generate long term and cost effective energy savings for Union s customers. DNV GL 11/10/16 Page 83

99 The Union Custom offering covers opportunities where energy savings are linked to unique building specifications, design concepts, processes and new technologies that are outside the scope of prescriptive and quasi prescriptive measures. The offering and incentives are targeted directly to the end user, while trade allies involved in the design, engineering and consulting communities assist to expand the message of energy efficiency. In addition to the Custom program, Union has launched the following programs as part of the C&I offering. Performance Based Conservation Pilot: launched in 2015, it is a data driven pilot for customer engagement, project identification and energy savings verification. The goals are to enroll up to 150 buildings, quantify energy savings (electric and gas) and water savings opportunities, identify highpotential buildings, determine and recommend conservation measures, and monitor and verify performance improvements. Energy Pathfinder Initiative: a project designed to explore, define, and quantify the opportunity for optimizing end-use operations and energy intensive processes, and to develop best practices for energy optimization and waste energy reduction within the Ontario manufacturing sector Key objectives The overall objectives for the Commercial and Industrial Custom offerings are straightforward - maximize cost-effective natural gas savings while pursuing deep energy savings. The scorecard metrics affecting the shareholder incentive are shown in Table Table scorecard metrics: Commercial and Industrial Custom Energy Solutions and Custom Offering Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Union Gas Cumulative Natural Gas Savings Cumulative Natural Gas Savings 1,214.1 million CCM/year (all resource acquisition programs) million CCM/year (C&I Custom 2016 target) million CCM (all large volume customers) million CCM (all small volume customers) million CCM (C&I Custom 2016 target) $3,227,419 $3,170,422 Source: Schedule B Decision and order OEB-approved scorecard weighting Gas Distribution, Inc. (EB ) / Union Gas Limited (EB ) January 20, 2016 DNV GL 11/10/16 Page 84

100 Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate and shareholder incentive amount. Table 4-18 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table Risks, impacts, and evaluation activities: Commercial and Industrial Custom Energy Solutions and Custom Offering Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Both Programs Jointly Inaccurate savings estimates High High Third party verification High Inaccurate baselines Medium High Market assessment Medium to High Inaccurate operating assumptions Poor base case documentation High High Third party verification High Medium/High Third party verification High Medium to High Inaccurate free ridership assumptions High High Participant survey Low Inaccurate spillover assumptions High Low Participant survey with desk reviews Medium Inaccurate measure life assumptions Medium High Persistence study High Inaccurate incremental cost assumptions Interconnection to C&I behavior programs Low Medium Participant survey Market assessment Medium Medium Third party verification Knowledge and understanding of C&I behavior projects Participant survey Low (participant survey) Medium (market assessment) Low DNV GL 11/10/16 Page 85

101 Interconnection to prescriptive and quasi-prescriptive measures Medium Medium Third party verification Knowledge and understanding of Prescriptive projects Low Electricity and water savings interactions may not be captured Medium Low Participant survey Third party verification High Evaluation activity recommendation This section describes the potential evaluation activities in greater detail with a description of the study objectives and a high-level outline of the key tasks. Although the evaluation activities are outlined as discrete tasks, they can be combined to produce more comprehensive studies. For example, the third party verification can be supplemented with participant surveys to provide a more streamlined evaluation that places less burden on the customer. For this program, the EC recommends completing annual NTG and verification evaluations which will include telephone surveys and selective on-site visits for large or uncertain projects. The EC also recommends a cross-cutting persistence study across a number of C&I programs Savings verification Program gross savings are currently based on estimates developed by utility representatives in conjunction with trade allies and program participants. The C&I Custom projects get individual savings estimates calculated for each project to reflect the specific equipment, operating conditions, and baseline conditions for that project. The program has traditionally used a third-party verification contractor, overseen by the utilities, to review the savings estimates and a third-party auditor to confirm the results of the verification contractor. Third party verification is the industry best-practice for confirming the energy savings realized through custom energy efficiency programs; however, we the EC recommends a single contractor overseen by the OEB rather than a contractor and an auditor. This evaluation activity addresses these risks from Table 4-18: Inaccurate savings estimates. Changes in installation details between incentive approval (and tracking savings estimate calculation) and the evaluation are common. The third-party verification will identify the discrepancies and determine a more accurate estimate that reflects the actual installation. Inaccurate operating assumptions. Changes in operation between incentive approval (and tracking savings estimate calculation) and the evaluation are common. The third-party verification will identify the discrepancies and determine a more accurate estimate that reflects the actual equipment operation. Poor base case documentation. A predominance of larger and process-related projects suggests a higher likelihood of uncertainty related to baselines. Program and project documentation is rarely organized to facilitate evaluation; it is typically configured to comply with program administration and incentive procedures. In cases where the base case documentation is not fully informative, the evaluation will incur additional uncertainties. The third-party verification can be used to fill in the blanks and ensure a more accurate savings estimate. DNV GL 11/10/16 Page 86

102 Interconnection to C&I behavior programs. The services offered by C&I behavior or market transformation programs like Run it Right, Run Smart, Comprehensive Energy Management, and Strategic Energy Management may interact with the energy savings from the projects that receive incentives under the C&I Custom program. Interconnection to prescriptive and quasi-prescriptive measures. Projects, as defined by the customer, may include prescriptive and custom measures from a program perspective. The verification study will need to be clear about what portion of the project each question is meant to address, and interactive effects between custom and prescriptive savings should be taken into account. In addition, the presence of custom and prescriptive measures at one site should cause the verification contractors to coordinate to minimize burden on the customer. Electricity and water savings interactions may not be captured. The installation of natural gas energy efficiency measures (such as low-flow showerheads) can result in electric or water savings which must be taken into account in the cost effectiveness calculation but may not be captured in program tracking or documentation. The third-party verification can be used to fill in the blanks and ensure a more accurate representation of program benefits. The objectives of the third party verification would be to: Verify that the energy efficiency measures were installed as reported and continue to operate as designed Verify the energy savings resulting from the installed energy efficiency measures Identify systematic program improvements that can increase the accuracy of program-produced energy savings estimates going forward. The third party verification should include the following tasks: Request and receive tracking data files and project documentation sufficient to identify the size and type of project installed as well as any services received through or projects promoted by the Run Smart, Strategic Energy Management, or C&I Prescriptive programs. Design a sophisticated sample and ensure that it represents the program and achieves a precision of 90/10. Conduct telephone calls, desk reviews, and on-site visits to verify installation and confirm the gross energy savings. Produce site-level reports that describe the measure, the savings calculation, the assumptions used in the calculation, the information found during the verification, and any changes to the energy savings estimate. Write a report that summarizes the results of the study and provides recommendations for improvement Participant surveys Program net savings are currently based on a stipulated NTG factor. A participant survey can use industrystandard data collection methods to identify the influence the program had on the participant s decision to install an energy efficient measure for a sample of program participants. This evaluation activity addresses these risks from Table 4-18: Inaccurate free ridership assumptions. Inaccurate free ridership assumptions will result in an incomplete understanding of the influence and value of the energy efficiency program. The participant survey will produce an accurate free ridership estimate specific to Ontario and the evaluated program year. DNV GL 11/10/16 Page 87

103 Inaccurate spillover assumptions. Inaccurate spillover assumptions will result in an incomplete understanding of the influence and value of the energy efficiency program. The participant survey will produce an accurate spillover estimate specific to Ontario and the evaluated program year. Inaccurate incremental measure cost assumptions. A participant survey can collect accurate information on the incremental cost related to custom projects, which will result in more accurate costbenefit study results. Interconnection to C&I behavior programs. The services offered by C&I behavior or market transformation programs like Run it Right, Run Smart, Comprehensive Energy Management, and Strategic Energy Management may influence the customer s decision to install projects under the C&I Custom program. A participant survey can capture that influence and accurately represent it in the NTG adjustment. Interconnection to prescriptive and quasi-prescriptive measures. Projects, as defined by the customer, may include prescriptive and custom measures from a program perspective. A participant survey can separate the two projects in the customer s mind and accurately identify the NTG adjustment specific to the C&I Custom program. The objectives of the participant survey would be to: Determine the free ridership and spillover associated with the installation of the energy efficiency measures. Gather information on incremental measure cost. Distinguish the effects of the C&I Custom program from the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, and C&I Prescriptive programs. Understand the influence that participation in the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, and C&I Prescriptive programs had on the customer s decision to install the energy efficiency measure. The participant survey should include the following tasks: Request and receive tracking data files and project documentation from and Union sufficient to identify the size and type of project installed as well as any services received through or projects promoted by the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, and C&I Prescriptive programs. Design a sophisticated sample and ensure that it represents the program and achieves a precision of 90/10. Write a telephone survey and deliver it to project decision-makers to determine the effect of the C&I Custom program on the company s decision to install an energy efficiency measure; particularly the effect on the timing, efficiency, and size of the measure. - Notify the interviewer of the services received through or projects promoted by the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, and C&I Prescriptive programs so the interviewer can properly account for them during the customer data collection. - Ask about internal policies on simple payback or rate of return for approving energy and non-energy projects. - Probe for non-energy benefits sold by the program when measuring influence. Identify measures that were installed without an incentive, but that could be as a result of the company s program participation (spillover). If spillover exists, conduct follow-up phone calls by DNV GL 11/10/16 Page 88

104 engineers to gather the data necessary to estimate the energy savings resulting from the measure installed. Conduct a ratio estimation analysis to expand the sample results to the population. Write a comprehensive report on the overall program free ridership and spillover, assembled into a representative NTG rate. Examine the free ridership and spillover to provide information that may help improve program design. Make recommendations to improve the quality of the program s energy savings calculations in future program years Market assessment Baseline assumptions for custom project energy savings estimates are currently determined on a case-bycase basis, but it is unclear what informs those selections. Market assessments can be done on a broad level for popular measures such as boilers or a case-by-case level for unique projects. This evaluation activity addresses these risks from Table 4-18: Inaccurate baselines. The baseline for custom projects may not be immediately identifiable, especially for innovative or process-related energy efficiency projects. A market assessment, whether done for a commonly installed project or on a case-by-case basis, can produce a more accurate understanding of standard practice and the appropriate calculation baseline. Inaccurate incremental measure cost assumptions. A market assessment can produce the average or typical incremental cost for commonly installed custom measures, which will result in more accurate cost-benefit study results. Using a market assessment for common measures could result in a lower cost and higher accuracy than a project-by-project study using participant surveys. The objectives of the market assessments would be to: Improve the cumulative savings by more accurately determining the measure baseline. Improve the cost-benefit analysis by more accurately determining incremental cost. The baseline portion of the market study should include the following tasks: For generalized baseline assumptions for a single measure across a number of installations: - Investigate the potential data available for study, such as sales data, government data, or program data. - Write a work plan to clearly identify the boundaries of the study: what technologies will be studied, what portions of the market will be measured, what will the outcome be. - Collect and analyze data according to the work plan. - Write a report on the methodology, results, and recommendations that may help improve program design. For a custom, single-measure baseline assumption at a single location (or for a single customer): - Identify market actors familiar with the process or equipment affected. - Conduct surveys with one or more market actors to collect data on the typical or standard practice installation under similar conditions. - Use the baseline to produce a more accurate energy savings estimate as part of the third-party verification. - Document the process, data collected, and result in the site-level report. The incremental cost portion of the market study should include the following tasks: DNV GL 11/10/16 Page 89

105 Investigate the potential data available for study, such as sales data, government data, or program data. Write a work plan to clearly identify the boundaries of the study, such as what technologies will be studied, what portions of the market will be measured, what will the outcome be. Collect and analyze data according to the work plan. Write a report on the methodology, results, and recommendations that may help improve program design Persistence study It is unclear how the measure life assumptions for custom projects are determined. In general, such assumptions are based on the mechanical life of the equipment installed (or replaced) but are not based on the actual operational life of the equipment. A persistence study can verify both assumptions and produce an accurate measure life assumption specific to Ontario. This evaluation activity addresses this risk from Table 4-18: Inaccurate measure life assumptions. A persistence study can identify how changes over the mechanical life of an installed measure can affect the energy savings realized by that measure, such as changes in operation practices, changes in building occupancy or tenant, and efficiency degradation. Section 5.4 contains additional information on persistence studies and how they are conducted Knowledge and understanding of other programs projects Union and offer a number of programs directed to the C&I sector in addition to C&I Custom, such as the C&I prescriptive program, Run it Right, Run Smart, Comprehensive Energy Management, and Strategic Energy Management. Although the utilities (and OEB and EC) generally treat the programs as distinct, customer participation in one program may influence their decision to participate in other programs, and customers may not be able to distinguish services from one program from services from another program. Knowing which customers participate in other programs and understanding the services they received and measures they installed provides a more complete picture which aids in data collection and spillover and free ridership estimates. This evaluation activity addresses these risks from Table 4-18: Interconnection to C&I behavior programs. The services offered by C&I behavior or market transformation programs like Run it Right, Run Smart, Comprehensive Energy Management, and Strategic Energy Management may influence the customer s decision to install projects and interact with the energy savings under the C&I Custom program. Interconnection to prescriptive and quasi-prescriptive measures. The services offered by the C&I Prescriptive program may influence the customer s decision to install projects and interact with the energy savings under the C&I Custom program. The objectives of knowing and understanding other programs projects would be to: Provide the best available information on customer participation to aid and improve accurate data collection. Identify how the various C&I programs interact and the effect of those interactions. Facilitate accurate C&I Custom program verification and NTG results. Knowing and understanding other programs projects should include the following tasks: DNV GL 11/10/16 Page 90

106 Request and receive tracking data files and project documentation sufficient to identify the services received through or projects promoted by the Run it Right, Run Smart, Comprehensive Energy Management, Strategic Energy Management, and C&I Prescriptive programs. Cross-reference the data to identify customers who participated in more than one program and identify how they participated and what services they received. Summarize the data for data collection and third party verification. Add a section to the third party verification and NTG reports that identify any interactions and their effects Small Commercial New Construction Program ( Pilot) Program Type Relative Size Resource Acquisition 0% of utility shareholder incentive ~1% of portfolio budget < 1% of portfolio savings Offered Since 2016 Past Evaluations No evaluation activities to date (new program) *Percentages are of the combined portfolio, inclusive of both utilities Program description is offering this program on a pilot basis in 2016, with results that will begin to be measured in The goal of this program is to provide small commercial builders and owners/developers access to cost effective energy modeling alternatives. 18 The program will provide financial incentives to implement energy performance modeling tools in addition to financial incentives for achieving specific energy efficiency targets. Participants will receive financial incentives for projects that achieve an energy efficiency level at least 5% above the current Ontario Building Code. 19 The target market includes builders and designers of new, Part 2 commercial buildings up to 75,000 square feet in s franchise area. The program will be evaluated at the mid-term to determine if it should be continued in 2019 and Key objectives The objectives of the Small Commercial New Construction program include overcoming the following barriers: 20 Increased cost of energy efficiency upgrades in a price-driven market Lack of funds, time, inclination, or knowledge of energy efficiency and related incentives Lack of training in the trades for installing energy efficiency measures. The scorecard metrics affecting the shareholder incentive are shown in Table EB , Exhibit B, Tab 2, Schedule 1 Page 30 of Section (pg. 22) of EB / EB Union Gas Limited and Gas Distribution Inc., Decision And Order, January 20, EB , Exhibit B, Tab 2, Schedule 1 Page 32 of 100 DNV GL 11/10/16 Page 91

107 Table /2017 scorecard metrics: Small Commercial New Construction Program Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Cumulative Natural Gas Savings million CCM (all 2016 large volume customers) million CCM (all 2016 small volume customers) 16.1 million CCM (Sm. Com. NC 2017 target) Unknown Source: Schedule B Decision and order OEB-approved scorecard weighting Gas Distribution, Inc. (EB ) / Union Gas Limited (EB ) January 20, 2016 The Small Commercial New Construction program does not have any scorecard metrics for Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate and shareholder incentive amount. Table 4-20 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table Risks, impacts, and evaluation activities: Small Commercial New Construction Program Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Pilot/ new program, needing disproportionate evaluation attention High TBD Impact evaluation Medium (phone interviews coupled with selective onsites) Evaluation activity recommendation This section describes the potential evaluation activities in greater detail with a description of the study objectives and a high-level outline of the key tasks. In order to mitigate the risks associated with administration and evaluation of this pilot program s performance, as stated in Table 4-20, the EC recommends completing an impact evaluation to include telephone surveys and selective on-sites for highly uncertain projects. This activity should not be completed until the program begins to realize energy savings. Given the low priority and the slow-developing nature of new construction programs, the EC recommends that the activities happen after the mid-term review. This evaluation activity addresses these risks from Table 4-20: Pilot/new program, needing disproportionate evaluation attention. The lack of prior evaluation induces the potential of over reliance on the TRM inputs and assumptions which may have been derived DNV GL 11/10/16 Page 92

108 from other jurisdictions, with differing climate and facility types. In addition to this, the lack of evaluation also suggests that there is no indication of program influence and free ridership issues. The objectives of the impact evaluation would be to: Verify that the energy efficiency measures were installed as reported and continue to operate as designed Verify the energy savings resulting from the installed energy efficiency measures Determine the free ridership and spillover associated with the installation of the energy efficiency measures. The impact evaluation should include the following tasks: Request and receive tracking data files and project documentation from sufficient to identify the size and type of project installed. Design a sophisticated sample and ensure that it represents the program and achieves a precision of 90/10. Write a telephone survey and deliver it to project decision-makers to determine the effect of the program on the decision to install an energy efficiency measure; particularly the effect on the timing, efficiency, and size of the measure. Identify measures that were installed without an incentive, but that could be as a result of the company s program participation (spillover). If spillover exists, conduct follow-up phone calls by engineers to gather the data necessary to estimate the energy savings resulting from the measure installed. Conduct desk reviews and on-site visits to verify installation and confirm the gross energy savings resulting from the measure installation. Conduct a ratio estimation analysis to expand the sample results to the population. Write a report on the methodology, results, and recommendations that may help improve program design Residential Savings by Design () Optimum Home (Union) Program Type Relative Size Market Transformation 3% of utility shareholder incentive ~5% of portfolio budget Offered Since 2012 Past Evaluations No evaluation activities to date *Percentages are of the combined portfolio, inclusive of both utilities Program description Both and Union offer similar residential market transformation programs that are focused on influencing builders and developers to construct new homes that exceed the efficiency requirements of the DNV GL 11/10/16 Page 93

109 Ontario Building Code standards. Through their programs, the utilities provide learning opportunities and financial assistance to residential developers throughout the building design process. It should be noted that this program offering focuses on total energy savings rather than natural gas savings only Key objectives The primary objective of this program offering is to demonstrate to builders the potential for achieving higher energy savings through proper building design. This in turn will increase the number of homes that are more efficient than required by the Ontario Building Code standards. The scorecard metrics affecting the shareholder incentive are shown in Table Table scorecard metrics: Residential Savings by Design and Optimum Home Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Number of Builders 33 Number of Homes Built 2,751 $313,500 Union Percentage of builders homes built to >20% above 2012 OBC 2015 Actuals + 20% $209, Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate (which is not part of the scorecard metrics) and shareholder incentive amount. Table 4-22 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. DNV GL 11/10/16 Page 94

110 Table Risks, impacts, and evaluation activities: Residential Savings by Design and Optimum Home Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Both Programs Jointly Actual impact of program activities is unknown Programs market effects are unknown High Low Verify program metrics through Desk review, participant survey, Site visits, or Building simulation High High Conduct a baseline study to determine program s likelihood to have measurable market effects. High (building simulation), Medium-High (site visits), Medium (participant survey), Low (desk review) High Evaluation activity recommendation For all market transformation programs, the potential evaluation activities include verification of program metrics and program market effects. In many cases, these activities can be conducted as discrete tasks or combined to produce a more comprehensive study of the program. For example, a participant survey could be leveraged to gather information on program metrics, program activity and to establish a market baseline. Below, the EC team describes in more detail each of the recommended evaluation activities for the market transformation programs. For the Residential Savings by Design and Optimum Home programs, the EC recommends telephone verification in Although the on-site visits will increase the appearance of rigor, the actual benefit received will be low unless a representative sample can be drawn, which requires all of the builders records. The building simulation would confirm the actual energy savings resulting from the programs, but since energy savings are not included in program metrics, the outcome will not affect the shareholder incentive. The EC recommends a baseline market effects study in 2017 (see section 5.1). The baseline study establishes the groundwork for measuring future market transformation by representing the pre-program state of the market. Future studies (2018 or beyond) would identify the change in the market and determine how much of that change is attributable to the utility programs Verification of program metrics Program impacts are currently represented solely by a desk review of the utility tracking data to verify the scorecard metrics. While this is sufficient to confirm the appropriate shareholder incentive relative to the utility scorecard, it does not fully capture the programs impacts, including the energy savings resulting from DNV GL 11/10/16 Page 95

111 the high efficiency homes built or the overall market transformation resulting from the program activities. This evaluation activity would produce a more rigorous verification of the scorecard metrics and could result in an estimate of the energy savings resulting from the homes built under the program. As outlined in EC s memorandum 2015 Verification Options for C&I Prescriptive and Market Transformation Programs (June 30, 2016), 21 there are several high-level options for verifying the installation or scorecard metrics of the residential new construction market transformation programs offered by Union and. These options can be combined to achieve a more robust verification study. The EC recommends building upon the results of any verification activity conducted on the 2015 programs in future years. For example, if a review of tracking data and phone survey of participants in 2015 indicates a high degree of uncertainty around the program, then the EC would recommend conducting on-site verification in Each of the verification activities are as follows: Desk Review. The verification contractor will summarize the electronic tracking data, including the total number of builders participating since program inception, the number of homes built by those builders, the number of new builders enrolled in the program year, and the number of homes built by new builders. The contractor will also review the HOT2000 verification methodology to determine whether site-specific models (i.e. matching the actual home that was built) were created or a prototype simulation was used with site-specific measures. Telephone Verification. The verification contractor will phone all of the builders who participated in the program to ask them how long they had participated, how many total homes have been built through the program since inception, and how many homes they built in the given program year. If the verification contractor is not able to reach all builders by phone, they will need to expand the results of the survey to the builder population. On-site Verification. This task starts with the activities performed in the telephone verification. In addition to participation information, the verification contractor will ask builders to provide the address and owner name for a sample of homes built under the program and a listing of the measures installed in those homes. The contractor will recruit homes for site visits and visually verify installation of the higher efficiency measures. Building Simulation. This task starts with the activities performed in the telephone verification and will only be conducted if the Desk Review identifies that prototype building simulations were used. In addition to participation information, the verification contractor will ask builders for blueprints and equipment specifications for a sample of homes built under the program and a listing of the measures that were installed to ensure the home exceeded building code. The verification contractor will create two models of each home using a standard building simulation program: one that meets code, and one that reflects the home as it was built. The simulations will be run to estimate annual building energy use and the difference calculated to determine whether the home achieves the desired improvement metric Market effects analysis To the best of our knowledge, to date there has not been a study to investigate awareness, knowledge, or attitudes toward above-code energy efficiency in the residential new construction market. The EC 21 See APPENDIX A. DNV GL 11/10/16 Page 96

112 recommends a holistic approach to determining market effects, including multiple studies to monitor change in the market over time and how much of that change can be attributed to the utility programs. The market effects research should include the following tasks: Conduct a baseline study in 2017 to measure builder and home buyer energy efficiency awareness and penetration, including knowledge of energy efficiency measures and building practices and the benefits of energy efficiency. This study will also identify the size of the program (and possible impact) relative to the overall new home market in and Union territory. If warranted, conduct periodic follow-up studies to measure the change in market practices with respect to energy efficient home construction and identify the portion of that change attributable to the utility programs. Future studies will take into account market effects caused by increasing levels of energy efficiency required by the Ontario Building Code over time Commercial Savings by Design Programs ( and Union) Program Type Relative Size Offered Since Past Evaluations Market Transformation 3% of utility shareholder incentive 2% of portfolio budget 2012 (); 2016 (Union) No evaluation activities to date *Percentages are of the combined portfolio, inclusive of both utilities Program description Similar to the Residential Savings By Design/Optimum Home programs, both and Union offer similar commercial market transformation programs that are focused on influencing builders and developers to construct new buildings that exceed the efficiency requirements of the Ontario Building Code standards. Through their programs, the utilities provide learning opportunities and financial assistance to commercial builders and developers throughout the building design process Key objectives The primary objective of this program offering is to demonstrate to builders the potential for achieving higher energy savings through proper building design. This in turn will increase the number of newly constructed commercial buildings that are more efficient than required by the Ontario Building Code standards. The scorecard metrics affecting the shareholder incentive are shown in Table DNV GL 11/10/16 Page 97

113 Table scorecard metrics: Commercial Savings by Design Programs Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Number of New Developments 33 $313,500 Union Number of New Developments 8 $209, Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate (which is not part of the scorecard metrics) and shareholder incentive amount. Table 4-24 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table Risks, impacts, and evaluation activities: Commercial Savings by Design Programs Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Both Programs Jointly Actual impact of program activities is unknown High Low Verify program metrics through: Desk review, Participant survey, Site visits, or Building simulation High (building simulation), Medium-High (site visits), Medium (participant survey), Low (desk review) Evaluation activity recommendation For all market transformation programs, the potential evaluation activities include verification of program metrics and program market effects. In many cases, these activities can be conducted as discrete tasks or combined to produce a more comprehensive study of the program. For example, a participant survey could be leveraged to gather information on program metrics, program activity and to establish a market baseline. Below, the EC team describes in more detail each of the recommended evaluation activities for the market transformation programs. Similar to the Residential Savings by Design and Optimum Home programs, the EC recommends telephone verification in 2017 for the Commercial Savings by Design Programs. Although on-site visits will increase the appearance of rigor, the actual benefit received will be low unless a representative sample can be drawn, DNV GL 11/10/16 Page 98

114 which requires all of the builders records. The building simulation would confirm the actual energy savings resulting from the programs, but since energy savings are not included in program metrics, the outcome will not affect the shareholder incentive. The EC recommends a baseline market effects study for the programs in 2017 (see section 5.1). A baseline study establishes the groundwork for measuring future market transformation by representing the preprogram state of the market. Future studies (2018 or beyond) would identify the change in the market and determine how much of that change is attributable to the utility programs Verification of program metrics Again, similar to the Residential Savings by Design/Optimum Home programs, the Commercial Savings by Design Program impacts are currently represented solely by a desk review of the utility tracking data to verify the scorecard metrics. While this is sufficient to confirm the appropriate shareholder incentive relative to the utility scorecard, it does not fully capture the programs impacts, including the energy savings resulting from the high efficiency homes built or the overall market transformation resulting from the program activities. This evaluation activity would produce a more rigorous verification of the scorecard metrics and could result in an estimate of the energy savings resulting from the homes built under the program. As outlined in EC s memorandum 2015 Verification Options for C&I Prescriptive and Market Transformation Programs (June 30, 2016), 22 there are several high-level options for verifying the installation or scorecard metrics of the commercial new construction market transformation programs offered by Union and. These options can be combined to achieve a more robust verification study. The EC recommends building upon the results of any verification activity conducted on the 2015 programs in future years. For example, if a review of tracking data and phone survey of participants in 2015 indicates a high degree of uncertainty around the program, then the EC would recommend conducting on-site verification in Each of the verification activities are as follows: Desk Review. The verification contractor will summarize the electronic tracking data, including the total number of builders participating in the program, the number of developments initiated by those builders, and the number completed. Telephone Verification. The verification contractor will phone all of the builders who participated in the program to ask them how long they had participated, how many total developments have been built through the program since inception, and how many developments were in progress or finished in the given program year. If the verification contractor is not able to reach all builders by phone, they will need to expand the results of the survey to the builder population. On-site Verification. This task starts with the activities performed in the telephone verification. In addition to participation information, the verification contractor will ask builders to provide the address and owner name for a sample of developments built under the program and a listing of the measures installed in those developments. The contractor will recruit developments for site visits and visually verify installation of the higher efficiency measures. 22 See APPENDIX A. DNV GL 11/10/16 Page 99

115 Building Simulation. This task starts with the activities performed in the telephone verification. In addition to participation information, the verification contractor will ask builders for blueprints and equipment specifications for a sample of developments built under the program and a listing of the measures that were installed to ensure the building exceeded building code. The verification contractor will create two models of each building using a standard building simulation program: one that meets code, and one that reflects the building as it was built. The simulations will be run to estimate annual building energy use and the difference calculated to determine whether the building achieves the desired improvement metric Market effects: baseline study Again, similar to the residential new construction market transformation programs, to the best of the EC s knowledge, there has not yet been a study of awareness, knowledge, or attitudes toward above-code energy efficiency. Therefore, we recommend a holistic approach to determining market effects, including multiple studies to monitor change in the market over time and how much of that change can be attributed to the utility programs. Initially, the market effects research should include the following tasks: Conduct a baseline study in 2017 to measure builder and building owner energy efficiency awareness and penetration, including knowledge of energy efficiency measures and building practices and the benefits of energy efficiency. This study will also identify the size of the program (and possible impact) relative to the overall C&I new construction market in and Union territory. If warranted, conduct periodic follow-up studies to measure the change in market practices with respect to energy efficient building construction and identify the portion of that change attributable to the utility programs Run it Right () RunSmart (Union) Program Type Relative Size Market Transformation 2% of utility shareholder incentive 2% of portfolio budget Offered Since 2016 (however similar programs have been offered since 2012) Past Evaluations No evaluation activities to date *Percentages are of the combined portfolio, inclusive of both utilities. It is necessary to report budget, savings and shareholder incentive information for both Union s Run Smart and SEM programs together as the detail provided to DNV GL at this time does not provide separate budgets for each program Program description Both and Union offer similar market transformation programs that are focused on improving operational efficiency among commercial customers. Through its program, provides customers with an energy assessment, technical and implementation assistance and performance monitoring, while Union offers customers incentives for undertaking low or no-cost energy improvements and activities in their facilities. DNV GL 11/10/16 Page 100

116 Key objectives The primary objective of these program offerings is to encourage enhanced energy performance among medium and large commercial and small industrial customers. By offering audits, technical assistance and monitoring, the program will increase building owners knowledge of available energy savings and encourage energy savings, while the Union program aims to have a similar result by improving overall building performance through the identification of low-cost building maintenance and operational changes. The scorecard metrics affecting the shareholder incentive are shown in Table Table scorecard metrics: Run It Right and Run Smart Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Participants 83 $253,637 Union Participants 28 $104, Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate and shareholder incentive amount. Table 4-26 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table Risks, impacts, and evaluation activities: Run It Right and Run Smart Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Both Programs Jointly Interconnection to C&I Custom, Prescriptive, and Large Volume Actual impact of program activities is unknown Medium Medium Include Run it Right / Run Smart effects in NTG studies High Low Verify program metrics through: Desk review, Participant survey, or Site visits. Low (NTG Interviews) Medium-High (site visits), Medium (participant survey), Low (desk review) DNV GL 11/10/16 Page 101

117 Evaluation activity recommendation For all market transformation programs, the potential evaluation activities include verification of program metrics and program market effects. In many cases, these activities can be conducted as discrete tasks or combined to produce a more comprehensive study of the program. For example, a participant survey could be leveraged to gather information on program metrics, program activity and to establish a market baseline. Below, the EC team describes in more detail each of the recommended evaluation activities for the market transformation programs. For the Run It Right/Run Smart programs, the EC recommends telephone verification in Although the on-site visits will increase the appearance of rigor, the cost to benefit ratio is high compared to telephone verification. The EC recommends that the Run it Right and Run Smart programs be considered and addressed in the annual verification and NTG study conducted on the Large Volume and C&I Custom programs. See sections and for more detail Verification of program metrics Similar to the other market transformation programs, the Run-It-Right/Run Smart program impacts are currently represented solely by a desk review of the utility tracking data to verify the scorecard metrics. While this is sufficient to confirm the appropriate shareholder incentive relative to the utility scorecard, it does not fully capture the programs impacts, including the energy savings resulting from the overall transformation of customer knowledge and awareness resulting from the program activities. This evaluation activity would produce a more rigorous verification of the scorecard metrics. There are several high-level options for verifying the installation or scorecard metrics of the Market Transformation programs offered by Union and. These options can be combined to achieve a more robust verification study. The EC recommends building upon the results of any verification activity conducted on the 2015 programs in future years. For example, if a review of tracking data and phone survey of participants in 2015 indicates a high degree of uncertainty around the program, then the EC would recommend conducting on-site verification in Each of the verification activities are as follows: Desk Review. The verification contractor will summarize the electronic tracking data, including the total number of customers participating in the program. Telephone Verification. The verification contractor will phone all of the customers who participated in the program to ask them about their participation in the program. If the verification contractor is not able to reach all customers by phone, they will need to expand the results of the survey to the builder population. On-site Verification. This task starts with the activities performed in the desk review. The contractor will recruit customers for site visits and visually confirm participation in the program Include Run it Right / Run Smart effects in NTG studies The Run it Right and Run Smart programs offer customers technical and financial assistance for improving the performance of their operations, including performance monitoring. Such services may lead to more actions for reducing energy use which may include energy efficiency measures that receive incentives DNV GL 11/10/16 Page 102

118 through the C&I Prescriptive, C&I Custom, and Large Volume programs, which will receive NTG evaluations in 2016 and Therefore, to properly understand the influence that the utilities have on the participants of the C&I Prescriptive and Custom and Large Volume programs, the NTG evaluators must understand the services provided to those customers through the Run it Right and Run Smart programs. Run it Right and Run Smart also have high spillover potential which should be attributed to the program where possible Comprehensive Energy Management () Strategic Energy Management (Union) Program Type Relative Size Market Transformation 1% of utility shareholder incentive 1% of portfolio budget Offered Since 2016 Past Evaluations No evaluation activities to date *Percentages are of the combined portfolio, inclusive of both utilities. This is CEM only; SEM budget was reported in the previous section because Union provided a combined budget for Run Smart and SEM programs. It is necessary to report budget, savings and shareholder incentive information for both Union s Run smart and SEM programs together as the detail provided to DNV GL at this time does not provide separate budgets for each program Program description Both and Union offer similar comprehensive market transformation programs targeting large commercial and industrial customers. These programs are focused on encouraging customers to monitor and analyze their own energy use by arming them with knowledge about the energy use in their buildings Key objectives The primary objective of this program offering is to provide energy use information to end-use customers in order to educate and inform customers of their own energy use patterns. This is turn should lead to the customer taking steps to reduce energy use. The scorecard metrics affecting the shareholder incentive are shown in Table Table scorecard metrics: Comprehensive Energy Management and Strategic Energy Management Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Number of Participants 7 $254,710 Union Number of Participants 3 Included in Run Smart incentive DNV GL 11/10/16 Page 103

119 Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate and shareholder incentive amount. Table 4-28 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table Risks, impacts, and evaluation activities: Comprehensive Energy Management and Strategic Energy Management Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Both Programs Jointly Interconnection to C&I Custom, Prescriptive, and Large Volume programs Actual impact of program activities is unknown No previous verification of program s market effects Medium Medium Include CEM/SEM effects in NTG studies High Low Verify program metrics through: Desk review, Participant survey, Site visits, or Building simulation High High Conduct a baseline study to determine program s likelihood to have measurable market effects. Low (NTG Interviews) Medium-High (site visits), Medium (participant survey), Low (desk review) High Evaluation activity recommendation For all market transformation programs, the potential evaluation activities include verification of program metrics and program market effects. In many cases, these activities can be conducted as discrete tasks or combined to produce a more comprehensive study of the program. For example, a participant survey could be leveraged to gather information on program metrics, program activity and to establish a market baseline. Below, the EC team describes in more detail each of the recommended evaluation activities for the market transformation programs. Similar to the Run it Right/Run Smart programs, the EC recommends telephone verification in 2016 for the CEM/SEM programs. The EC recommends a baseline market effects study for the programs be conducted in 2018 (see section 5.1). A baseline study establishes the groundwork for measuring future market transformation by DNV GL 11/10/16 Page 104

120 representing the pre-program state of the market. Future studies (2019 or beyond) would identify the change in the market and determine how much of that change is attributable to the utility programs. The EC recommends that the CEM/SEM programs be considered and addressed in the annual verification and NTG study conducted on the Large Volume and C&I Custom programs. See sections and for more detail Include CEM/SEM effects in NTG studies The CEM/SEM programs are focused on encouraging customers to monitor and analyze their own energy use by arming them with knowledge about the energy use in their buildings. The increased knowledge should lead to more actions for reducing energy use, which may include energy efficiency measures that receive incentives through the C&I prescriptive, C&I Custom, and Large Volume programs, which will receive NTG evaluations in 2016 and Therefore, to properly understand the influence that the utilities have on the participants of the C&I Prescriptive and Custom and Large Volume programs, the NTG evaluators must understand the services provided to those customers through the CEM and SEM programs. CEM and SEM also have high spillover potential which should be attributed to the program where possible Verification of program metrics Program impacts are currently represented solely by a desk review of the utility tracking data to verify the scorecard metrics. While this is sufficient to confirm the appropriate shareholder incentive relative to the utility scorecard, it does not fully capture the programs impacts, including the overall market transformation resulting from the program activities. This evaluation activity would produce a more rigorous verification of the scorecard metrics. There are several high-level options for verifying the installation or scorecard metrics of the Market Transformation programs offered by Union and. These options can be combined to achieve a more robust verification study. The EC recommends building upon the results of any verification activity conducted on the 2015 programs in future years. For example, if a review of tracking data and phone survey of participants in 2015 indicates a high degree of uncertainty around the program, then the EC would recommend conducting on-site verification in Each of the verification activities are as follows: Desk Review. The verification contractor will summarize the electronic tracking data, including the total number of customers participating in the program. Telephone Verification. The verification contractor will phone all of the customers who participated in the program to ask them about their participation in the program. If the verification contractor is not able to reach all customers by phone, they will need to expand the results of the survey to the builder population. On-site Verification. This task starts with the activities performed in the desk review. The contractor will recruit customers for site visits and visually confirm participation in the program Market effects: baseline study The EC recommends a holistic approach to determining market effects, including multiple studies to monitor change in the market over time and how much of that change can be attributed to the utility programs. The market effects research should include the following tasks: DNV GL 11/10/16 Page 105

121 Conduct a baseline study in 2017 to measure C&I customer energy efficiency awareness and penetration, including knowledge of energy efficiency measures and building energy management practices and the benefits of energy efficiency. This study will also identify the size of the program (and possible impact) relative to the overall existing C&I building market in and Union territory. If warranted, conduct periodic follow-up studies to measure the change in market practices with respect to building energy management and identify the portion of that change attributable to the utility programs School Energy Competition () Program Type Relative Size Market Transformation 1% of utility shareholder incentive ~1% of portfolio budget Offered Since 2016 Past Evaluations No evaluation activities to date (new program) *Percentages are of the combined portfolio, inclusive of both utilities Program description This program aims to educate and empower students to take energy saving action in their homes, schools and communities. does this through offering educational material and workshops to all grade levels as well as sponsoring an annual energy savings competition for students in grades Furthermore, makes information on each school s gas consumption and baseline usage available via a web-based data dashboard Key objectives The primary goal of the program is to increase knowledge among primary and secondary schools and drive energy savings within participating schools via friendly competition. The scorecard metrics affecting the shareholder incentive are shown in Table Table scorecard metrics: School Energy Competition Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ Number of Schools 55 $125, Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate (which is not part of the scorecard metrics) and shareholder incentive amount. Table 4-30 lists the DNV GL 11/10/16 Page 106

122 risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table Risks, impacts, and evaluation activities: School Energy Competition Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Actual impact of program activities is unknown High Low-Medium Desk review of tracking data, Telephone verification with program participants, On-site verification of a sample of participating schools, Building on telephone verification activity Low (desk review), Medium (telephone verification) Medium-High (onsite) Evaluation activity recommendation For all market transformation programs, the potential evaluation activities include verification of program metrics and program market effects. In many cases, these activities can be conducted as discrete tasks or combined to produce a more comprehensive study of the program. For example, a participant survey could be leveraged to gather information on program metrics, program activity and to establish a market baseline. Below, the EC team describes in more detail each of the recommended evaluation activities for the market transformation programs. For the School Energy Competition program, the EC recommends telephone verification to confirm program participation and verify the scorecard metric. Although the on-site visits will increase the appearance of rigor, the measures implemented during the competition are more likely to be behavior-based than equipment-based, which will not leave anything on site to view. Further, a billing analysis might uncover energy savings if the savings are large enough to exceed the typical variability of billing data (unlikely in this situation) but energy savings are not included in program metrics, therefore the outcome would not affect program metrics. Given the low priority of this program, the EC recommends that the verification happen after the mid-term review Verification of program metrics There are several high-level options for verifying the installation or scorecard metrics of the Market Transformation programs offered by Union and. These options can be combined to achieve a more robust verification study. The EC recommends conducting verification activities for all market transformation programs. Each of the verification activities are as follows: DNV GL 11/10/16 Page 107

123 Desk Review. The verification contractor will summarize the electronic tracking data, including the total number of schools participating in the program. Telephone Verification. The verification contractor will phone all of the schools who participated in the program to ask them about their participation. If the verification contractor is not able to reach all schools by phone, they will need to expand the results of the survey to the school population. On-site Verification. This task starts with the activities performed in the telephone verification. If the measures installed during program participation include equipment replacement, the contractor will recruit schools for site visits and visually confirm participation in the program Energy Literacy Program () Program Type Relative Size Market Transformation 0% of utility shareholder incentive ~1% of portfolio budget Offered Since 2017 Past Evaluations No evaluation activities to date (new program) *Percentages are of the combined portfolio, inclusive of both utilities Program description The Energy Literacy program (or initiative, as it has no associated shareholder incentive) provides umbrella marketing activities throughout the year by engaging customers through a myriad of marketing activities including interactive booths at shopping malls and video games for younger audiences. It should be noted that this program focuses on both gas and electricity conservation programs in Ontario Key objectives The primary objective of this program is to increase the energy literacy of customers, including their ability to: Trace energy flows and think in terms of energy systems Know how much energy they use, for what purpose, and where the energy comes from Assess the credibility of information about energy Communicate about energy and energy use in meaningful ways Make informed energy use decisions based on an understanding of impacts and consequences. 23 The offering theory is as follows: In the short term, the offering elements will increase customers energy literacy, including their understanding of energy and its impact on their lives. In the medium term, customers will take greater control of their energy use and energy policy. In the long term, informed customers will use energy wisely and support public initiatives that do the same. 23 Adapted from DNV GL 11/10/16 Page 108

124 This offering does not have an associated shareholder incentive, as shown in Table Table scorecard metrics: Energy Literacy Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ None NA NA Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate, which is not part of the scorecard metrics. Table 4-32 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table Risks, impacts, and evaluation activities: Energy Literacy Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost The effect of the program is unknown High Low General population survey Medium Evaluation activity recommendation For all market transformation programs, the potential evaluation activities include verification of program metrics and program market effects. In many cases, these activities can be conducted as discrete tasks or combined to produce a more comprehensive study of the program. For example, a participant survey could be leveraged to gather information on program metrics, program activity and to establish a market baseline. Below, the EC team describes in more detail each of the possible evaluation activities for the market transformation programs. For the Energy Literacy program, the EC does not recommend any evaluation activities. There is no shareholder incentive associated with this program and no scorecard metrics to verify. Although program effects might be measured through a general population survey to gauge the change in energy literacy in territory, it would be difficult to translate that literacy into energy savings values or other market effects that are not already being captured through other program evaluation studies General population survey The objectives of a general population survey would be to measure the change in energy literacy customers and determine the influence of the Energy Literacy program on that change. A general population study should include the following tasks: Desk Review. The verification contractor will summarize the program s activities including the method and frequency of each activity. DNV GL 11/10/16 Page 109

125 Telephone Verification. The verification contractor will conduct a general population survey of customers to ask them about their exposure to the program. The verification contractor will need to expand the results of the survey to the population Energy Leaders Pilot Program () Program Type Relative Size Market Transformation 0% of utility shareholder incentive < 1% of portfolio budget Offered Since 2016 Past Evaluations No evaluation activities to date (new program) *Percentages are of the combined portfolio, inclusive of both utilities Program description This new program (or initiative, as it has no associated shareholder incentive) is targeted to assist those commercial and industrial customers who have been identified as energy efficiency leaders to become even more efficient. This program differs from s other commercial and industrial programs as it has been tailored to target and assist those customers that have already undertaken basic efficiency upgrades and are looking to achieve harder-to-reach savings. Participants are identified through their participation in existing programs, such as Run it Right, Comprehensive Energy Management, and the C&I Custom program. Participants can also self-identify and receive site assessments and benchmarking that will confirm that they qualify as an Energy Leader. will offer Energy Leaders energy audits and increased customer incentives compared to those available in its other commercial and industrial programs to help customers implement new and innovative technologies Key objectives The primary goal of this pilot is to encourage repeat program participation among commercial and industrial customers who have already completed the basic energy efficiency projects available to them. This offering does not have an associated shareholder incentive, as shown in Table Table scorecard metrics: Energy Leaders Pilot Program Metric Target (100%) Maximum Shareholder Incentive (if 150% of target achieved), $ None NA NA 24 Modified from EB / EB DNV GL 11/10/16 Page 110

126 Program risks, impacts, and targeted evaluation activities There are a number of uncertainties (risks) that threaten the accuracy of the programs natural gas savings estimate (which is not part of the scorecard metrics) and shareholder incentive amount. Table 4-34 lists the risk, the likelihood that the risk will occur, the potential impact of the risk, and the evaluation activity and relative cost to address the risk. Table Risks, impacts, and evaluation activities: Energy Leaders Pilot Program Risk Description Likelihood Impact Evaluation Activity Relative Evaluation Cost Actual impact of program activities is unknown High Low NTG comparison Low-Medium Evaluation activity recommendation For all market transformation programs, the potential evaluation activities include verification of program metrics and program market effects. In many cases, these activities can be conducted as discrete tasks or combined to produce a more comprehensive study of the program. For example, a participant survey could be leveraged to gather information on program metrics, program activity and to establish a market baseline. Below, the EC team describes in more detail each of the possible evaluation activities for the market transformation programs. For the Energy Leaders program, the EC does not recommend any evaluation activities. There is no shareholder incentive associated with this program and no scorecard metrics to verify. The energy savings resulting from program participation are already captured through participation in other programs. The influence of the program would presumably be reflected in lower free ridership rates for Energy Leader participants when compared to other program participants; at most, the EC recommends a side-study attached to an existing NTG analysis that would fund an oversample of Energy Leader participants in the NTG study and a post-study analysis to compare the Energy Leader free ridership results to those of non- Energy Leader participants NTG comparison The objectives of the NTG comparison would be to measure the free ridership of the Energy Leader participants at a statistically viable level and compare the results to non-energy Leader participants to identify whether the additional incentives and services resulted in overcoming additional barriers. The comparison should include the following tasks within a larger NTG study: Identify the Energy Leaders in the existing NTG study Oversample Energy Leader participants to ensure a statistically viable result from the Energy Leader group Compare the resulting Energy Leader free ridership to non-energy Leader free ridership to identify whether the additional incentives and services resulted in a higher rate of adoption than in the general program population. DNV GL 11/10/16 Page 111

127 5 CROSS-CUTTING CONSIDERATIONS In any portfolio of energy efficiency programs there are issues which cut across programs. Often these are related to specific sectors, markets or end-uses which are touched by multiple program activities. Taking a holistic approach to both the planning and evaluation of a portfolio of programs versus planning evaluations of each program individually allows the OEB to consider these cross-cutting issues and plan for them in such a way that often leads to greater efficiency in evaluation spending and more robust and rich data collection while reducing response burden for individual utility customers. For example, the EC has grouped the residential home retrofit programs with the low-income single-family programs into a single evaluation study because the programs are similar and the evaluations follow very similar activities. The C&I Prescriptive program and the C&I Direct Install program are very different in design and implementation; as a result, the evaluation plans for each are very different. However, they share many of the same measures, so an evaluation specific to those measures (such as low-flow showerheads) would be considered cross-cutting. The following sections present several cross-cutting issues specific to the natural gas DSM portfolio in Ontario. 5.1 Market assessment studies It is often anticipated that market transformation programs will produce market effects including increased awareness, knowledge and adoption of efficient technologies and/or energy management practices which can lead to measurable savings. Yet these effects can be difficult to measure in new or smaller program initiatives. Conversely, larger programs, particularly those that offer incentives and have associated energy savings, often induce market effects that can be effectively measured. These studies can be relatively expensive and may not always produce a satisfactory result. Much depends on the amount of information available and the cooperation of the market actors, such as builders, equipment distributors, or participants, and their ability to accurately reflect sometimes unknown levels of energy efficiency awareness and previous activities. The primary considerations for a program s candidacy for market effects research are: The program s expected energy savings The size of the program The age of the program The timing of significant program changes As such, the EC recommends that market assessment studies for market transformation programs be conducted on the two programs most likely to induce measurable energy savings: Commercial Savings by Design; Residential Savings by Design/Optimum Home. Market effects studies of other commercial and residential new construction programs in North America, particularly those with a similar climate to Ontario, reveal that these programs are good candidates for generating measurable market effects. These programs are expected to grow significantly in the next couple of years, making 2016 an ideal year to study the baseline. As such we recommend conducting the baseline research in Strategic Energy Management/Comprehensive Energy Management. Market effects studies have been conducted on similar programs in the Pacific Northwest, and have revealed that these types of programs can and do have measurable market effects. Further, the program is in its first year which is DNV GL 11/10/16 Page 112

128 an ideal time to conduct baseline research. However, given the relative size of the CEM/SEM initiatives compared to the new construction programs, the EC recommends delaying this research to 2018 (prior to the mid-term review), unless the available budget for market effects work can accommodate a baseline study of the SEM/CEM initiatives as well The EC does not recommend studying market effects for the remaining market transformation programs, including the Run-it-Right/Run Smart initiatives, given lower probability of measurable savings from those efforts as well as the relatively small budgets and shareholder incentives associated with the programs. 5.2 Coordination with other research and evaluation activities As noted above, addressing cross-cutting issues upfront in the evaluation planning stages often leads to significant benefits including the ability to coordinate stakeholder interests, data collection activities, program analyses and methods development. Furthermore, it can generate greater value for the evaluation budget since a coordinated effort will reduce inefficiencies in data collection activities and reduce respondent burden. For example, a planned data collection for a current impact study can be leveraged to collect additional data re: the market and customer experience to be analysed under a future process or market assessment study. Any coordination efforts should happen both across the evaluation planning activities of the OEB and also with other entities planning evaluation and research activities being undertaken by Union, and the IESO that could benefit from leveraging methods development, data collection activities and program analyses. Based on our current understanding of the planned research and evaluation activities in 2016, 2017 and 2018, the EC has identified the following areas worthy of consideration for a coordinated approach across research and evaluation activities of both the OEB and other stakeholders. 5.3 Process evaluation The EC understands that process evaluation has traditionally been the purview of the gas utilities as program implementers. However, it is often difficult to fully understand the results of an impact evaluation without the context provided by a process evaluation. In addition, process and impact evaluations typically collect data from the same group of participants, and conducting multiple studies evaluating the same group may result in confusion and respondent fatigue. Any planned process evaluation studies could be informed by virtually all of the planned data collection activities discussed throughout this EM&V Plan. These include the Program Materials Review, Program Tracking Data review, Program Administrator (PA) interviews, Verification and Due Diligence Review, and Participant Telephone Surveys. Each of these data collection activities will provide qualitative and quantitative information on the program s structure, design, and performance in comparison with intentions, customer experience, or a combination of these factors. Therefore, the EC recommends working with the gas utilities and IESO as part of a process evaluation working group to take the following steps: Develop a Process Evaluation Roadmap that will catalog all currently planned process evaluation activities and identify the key areas for coordination of methods development, data collection and analyses between the planned process evaluation efforts and the EC recommended verification and market effects evaluation contained herein. Through working group meetings with the utilities and the IESO (and other interested stakeholders), the EC will catalog the planned process evaluation studies for the current evaluation period ( ). If desired by the working group, the EC will also recommend additional process evaluation research based on the same value of information framework leveraged herein. DNV GL 11/10/16 Page 113

129 Leverage planned customer touch points to collect information on program effectiveness and perception. Once the planned process evaluation studies are identified and cataloged in the roadmap, the EC will work with the entity leading the study (e.g. Union,, IESO or other) to ensure that any OEB-led studies focused on the same customer and/or market actor group(s) are also collecting data on program effectiveness and perception that can then be leveraged by the process evaluation study team. For example, if is leading a process evaluation of the Direct Install program, the verification contractor can collect both installation data and program effectiveness and perception data during the same customer touch point. The program effectiveness and perception data can then be passed to the process evaluation study lead for use in their own evaluation analysis. Facilitate coordination of methods development and evaluation analyses. Similar to the process outlined about related to data collection coordination, once the planned process evaluation studies are identified and cataloged in the roadmap, the EC will work with the entity leading the study to ensure that coordination occurs anywhere that evaluation methods or analyses should be aligned. 5.4 Persistence studies Utility energy savings goals are established in cumulative cubic meters of natural gas, which reflect the natural gas saved over the entire installation period of the measure. However, it is industry standard practice to calculate a savings estimate based on the first year of measure installation and multiply it by a stipulated measure life, essentially assuming that the same energy savings will accrue annually until an average life is achieved. Persistence studies are conducted to determine the average annual life for a particular measure, often called the effective useful life (EUL) of the measure. The studies typically address: How long a measure is kept in place relative to its useful life The impact of technical degradation The impact of operating conditions Persistence studies are generally planned to produce EUL estimates at a measure and sector level. For example, a separate EUL would be collected for a commercial boiler from an industrial boiler and for a residential furnace from a commercial furnace. Given the number of sector and measure combinations and the desire for each to achieve a certain level of precision, it is necessary to prioritize the areas that can be best improved by applying persistence study funds. The EC recommends the following activities for a persistence study: Develop a Persistence Study Roadmap that will identify areas of persistence uncertainty and the impact those areas have on current and past program CCM savings. The roadmap must consider: - The degree to which the measure and sector combination have been studied in other jurisdictions and the applicability of those results to the Ontario portfolio - Whether current program measures are likely to continue in future program designs - The impact of past program measures on the assumptions currently used for load and distribution forecasting and other planning activities. Similar to this evaluation plan, the resulting roadmap should identify the persistence activities that should be undertaken and prioritize them for future study. DNV GL 11/10/16 Page 114

130 Establish a standard persistence study methodology for use in Ontario which could address both natural gas and electric program measures. By developing a standard methodology, the EC ensures that persistence studies result in comparable EUL estimates across study years. In general, we recommend a high-level study methodology that includes the following steps: - Identify a representative sample of prior program implementations of the measure in the sector that is being addressed. - Conduct a telephone survey or on-site visit to: Determine whether the measure is still installed Identify the current efficiency level relative to the initial installation or other objective expectations Confirm that the measure is currently operating and how it is being utilized relative to initial expectations - Using the measure-level information, determine the EUL for the measure and sector combination being studied and identify trends that may suggest short-term Band-Aid adjustments to related technologies or sectors until they can be studied. Identify areas for integration with ongoing verification and NTG studies. To increase evaluation efficiency, it is important to plan persistence activities in conjunction with other data collection efforts. For example, a past measure installed at Customer Site A might be identified for participation in the persistence study in the same year that a current measure installed at Customer Site A is selected for a verification visit. Conduct annual persistence studies that address measure and sector combinations according to the priorities identified in the roadmap. Incorporate the results of the studies into the TRM and custom program calculations. DNV GL 11/10/16 Page 115

131 APPENDIX A. LIST OF FIGURES AND TABLES List of figures Figure 2-1. Distribution of combined planned or achieved cumulative savings and budget, savings, and total available or achieved shareholder incentive across program categories Figure resource acquisition programs Figure Union resource acquisition programs Figure Combined resource acquisition programs Figure Union Large Volume program Figure low income programs Figure Union low income programs Figure Combined low income programs Figure market transformation programs Figure Union market transformation programs Figure Combined market transformation programs List of tables Table 1-1. Programs approved for 2015, 2016, and Table 1-2. Steps in completing the value of information exercise... 3 Table 2-1. Planned or achieved cumulative savings and budget and maximum available or achieved shareholder incentive... 8 Table 2-2. Distribution of planned or achieved cumulative savings and budget and total available or achieved shareholder incentive across program categories... 9 Table 2-3. Distribution of 2015 budget, savings, and shareholder incentive across programs for Union,, and combined Table 2-4. Distribution of 2016 budget, savings, and shareholder incentive across programs for Union,, and combined Table 2-5. Distribution of 2017 budget, savings, and shareholder incentive across programs for Union,, and combined Table 2-6. Program maturity, similarity to electricity CDM offerings, and previous evaluations Table 2-7. EC-Recommended program prioritization: portfolio analysis Table 3-1. Summary of the evaluations of the 2015 program year portfolio Table scorecard metrics: Home Energy Conservation and Home Reno Rebate Table 4-2. Risks, impacts, and evaluation activities: Home Energy Conservation and Home Reno Rebate Table scorecard metrics: Residential Adaptive Thermostats Table 4-4. Risks, impacts, and evaluation activities: Residential Adaptive Thermostats Table scorecard metrics: Home Winterproofing, Home Weatherization, Furnace End-of-Life, and Aboriginal Offerings Table 4-6. Risks, impacts, and evaluation activities: Home Winterproofing, Home Weatherization, Furnace End-of-Life, and Aboriginal Offerings Table scorecard metrics: Low Income Multi Residential Affordable Housing and Low-income Multifamily Offering Table 4-8. Risks, impacts, and evaluation activities: Low Income Multi Residential Affordable Housing and Low Income Multi-Family Offering Table scorecard metrics: Low Income New Construction Table Risks, impacts, and evaluation activities: Low Income New Construction Table scorecard metrics: Large Volume Table Risks, impacts, and evaluation activities: Large Volume Table scorecard metrics: Commercial and Industrial Prescriptive Offerings Table Risks, impacts, and evaluation activities: Commercial and Industrial Prescriptive Offerings Table scorecard metrics: Commercial and Industrial Direct Install Programs Table Risks, impacts, and evaluation activities: Commercial and Industrial Direct Install Programs Table scorecard metrics: Commercial and Industrial Custom Energy Solutions and Custom Offering DNV GL 11/10/16 Page 116

132 Table Risks, impacts, and evaluation activities: Commercial and Industrial Custom Energy Solutions and Custom Offering Table /2017 scorecard metrics: Small Commercial New Construction Program Table Risks, impacts, and evaluation activities: Small Commercial New Construction Program Table scorecard metrics: Residential Savings by Design and Optimum Home Table Risks, impacts, and evaluation activities: Residential Savings by Design and Optimum Home.. 95 Table scorecard metrics: Commercial Savings by Design Programs Table Risks, impacts, and evaluation activities: Commercial Savings by Design Programs Table scorecard metrics: Run It Right and Run Smart Table Risks, impacts, and evaluation activities: Run It Right and Run Smart Table scorecard metrics: Comprehensive Energy Management and Strategic Energy Management Table Risks, impacts, and evaluation activities: Comprehensive Energy Management and Strategic Energy Management Table scorecard metrics: School Energy Competition Table Risks, impacts, and evaluation activities: School Energy Competition Table scorecard metrics: Energy Literacy Table Risks, impacts, and evaluation activities: Energy Literacy Table scorecard metrics: Energy Leaders Pilot Program Table Risks, impacts, and evaluation activities: Energy Leaders Pilot Program DNV GL 11/10/16 Page 117

133 APPENDIX B EVALUATION MEMOS Memo to: Natural Gas Demand Side Management Evaluation Advisory Committee (EAC) From: DNV GL-Energy Date: June 6, 2016 Prep. by: Tammy Kuiken 2015 Custom Program Savings Verification (CPSV) Proposal This memo presents a proposal for completing the 2015 CPSV integrated with the Custom Program Net to Gross (NTG) study already in process with DNV GL. The advantages of integrating the two studies include: Increased efficiency, resulting in a lower overall cost - One sampling process that can be modified to target the needs of both studies - One recruiting process - One documentation review to familiarize the evaluation team with the project installed The ability to use the life cycle net savings method for the NTG study (depending on the extent of the CPSV effort) A more comprehensive, overarching analysis of the results which will produce more effective recommendations and improved programs. The Evaluation Contractor (EC) team has outlined the CPSV approach employed in 2014 as well as the suggested CPSV approach for The suggested approach for 2015 will include a larger number of verified projects, more sophisticated sampling, more rigorous site-level measurement and verification (M&V), and more comprehensive analysis. Previous Process and proposed enhancements The EC team has reviewed the major components of the previous verification approach and has summarized its observations below, including the general enhancements it is suggesting for the 2015 verification work and the pros and cons of the proposed alterations. For reference, the following table shows the number of sites that were sampled in the previous approach. Previous Approach (2014) Program Segment N n Union - C&I Union - Large Volume Union Multi-Family TBD 10 Sub-Total - UNION Commercial Industrial Multi-Family TBD 16 Sub-Total - ENBRIDGE TOTAL All DNV GL 11/10/16 Page 118

134 Previous Process Proposed Enhancement Pros to Enhancement Sampling: The sampling process was developed by Navigant for a typical custom program, not the specific projects in the program year being verified. It was stratified by utility, program, and size. Site-level rigour: A similar level of rigour was applied to all sites selected for CPSV. On-sites: There were 102 onsites conducted on the 2014 program year. Post-Collection Analysis: Many of the verifier and auditor resources were concentrated on confirming the savings verified through the CPSV. Any postcollection analysis was completed by the utilities. We propose a more sophisticated sample design which would include calculation type/measure type in addition to utility, program, and size. It is also coordinated with the NTG sampling. We propose a range of rigour options applied to more closely match the needs of the project being reviewed. We propose to review 184 projects. 112 would be on-sites and 72 would be Telephone Supported Engineering Review (TSERs). We propose to apply analysis time to identify big-picture themes that relate to program performance improvement and that could also improve accuracy of savings calculations. Allows for a more efficient sample, which provides greater precision for a similar number of sites. Facilitates better analysis after the data collection is done. Overall, allows for more projects to be reviewed and more efficiently spends the evaluation budget where it can make the most difference. This is a more rigorous process, with more projects reviewed and more on-sites completed. Enhanced stakeholder involvement up-front in research issue definition and evaluation planning decision-making lessens concerns following CPSV completion. Deliberate and transparent focus on program performance metrics and improvement, while maintaining or improving accuracy of final CSPV savings estimates. Proposed 2015 CPSV approach The EC-proposed approach for the 2015 CPSV process increases the number of reviewed projects, adds gradations of rigour for the project reviews, and includes a comprehensive post-collection review across all verified projects and the NTG effort. The following table describes the gradations of rigour applied to the CPSV reviews. The rigour increases as one moves down the table; similarly the proportion of Itron (and/or DNV GL) engineering staff compared to Stantec engineering staff conducting the review tasks also increases. Based on our review, the proposed Standard On-site seems similar to the level of effort used in past verification efforts. Travel costs (and time) will be embedded into the budget of each review category, except, of course, the TSER. DNV GL 11/10/16 Page 119

135 Proposed 2015 CPSV Approach Program Segment N n 25 Union - C&I and Multi-Family Standard On-Sites Union - C&I and Multi-Family TSERs 29 Union - Large Volume High Rigour On-Sites Union - C&I and/or Large Volume Very High Rigour Included above 4 Sub-Total UNION Commercial and Multi-Family Standard On-Sites Commercial and Multi-Family - TSERs 20 - Industrial Standard On-Sites Industrial TSERs 23 Commercial and/or Industrial) Very High Rigour Included above 2 Sub-Total ENBRIDGE TOTAL All Type of Review Review Activities Firm(s) Conducting Reviews Telephonesupported engineering review (TSER) Review files Determine CPSV rigour By phone, verify installation and confirm operating parameters Using participant data, confirm calculations and assumptions Compare verified savings to file Compare verified savings to results from other jurisdictions Itron (and/or DNV GL), Stantec Standard On-site Conduct all TSER components Develop a site-specific M&V plan to organize data collection in advance of on-site Conduct on-site visit of approximately 2 hours Conduct more extensive data collection; possibly on-site, possibly by phone Analyze and model savings based on data collection results Complete site report, adding flags for additional senior staff review Conduct an average of 1 hour of quality control by a senior Itron (and/or DNV GL), Stantec 25 The total number of site visits per program were determined using an assumption of 0.5 error ratio and the following stratifications: utility/program ( commercial and multi-family, industrial, Union commercial and industrial, Union large volume, Union multifamily), size (the smallest third of the program by savings, the largest third of the program by savings, and the remaining). DNV GL 11/10/16 Page 120

136 Type of Review Review Activities Firm(s) Conducting Reviews engineer High Rigour Onsite Very High Rigour / Case Study Conduct all TSER and commercial/light industrial on-site components Increase the rigour of the file review to explore connections to past projects, other equipment, and facility operations Increase the sophistication of the M&V plan to include the additional knowledge from the file review and data from NTG survey Double the length of the site visit to four hours, including an in-depth interview with operations or facility management Compare results of operations survey with NTG survey from decision-makers Follow up on additional issues raised by the greater depth of review Double the quality control effort by senior engineer to 2 hours Conduct all heavy industrial on-site components Increase the sophistication of the M&V plan and use more senior engineers, which must be carefully organized to minimize inconvenience to customers Conduct pre-visit phone calls to clarify program documentation and plan site visit Double the length of the site visit to 4 hours and use senior engineers Conduct additional in-depth interviews as necessary to cover all specialized knowledge Increase rigour of analysis and modelling, including indepth review of industry-standard baselines for nonstandardized equipment. Use senior engineers for reporting Triple the quality control effort by senior engineer to 3 hours Itron (and/or DNV GL), Stantec Itron (and/or DNV GL), Stantec The EC team selected a sample based on utility, program type, and size. Using our experience with similar programs in other jurisdictions, we distributed the selected sample across the rigour levels. Our proposed approach assumes 72 TSERs, 83 standard on-sites, 23 high rigour on-sites, and 6 very high rigour onsites/case studies, for a total of 112 on-sites and an additional 72 TSERs. The EC presents the proposed 2015 CPSV approach in the table below. The distribution of sample across the program segments should be viewed in a general sense and is subject to minor changes once the full DNV GL 11/10/16 Page 121

137 project files have been reviewed and input and advice has been provided on where evaluation funds and resources are most appropriately used. 26 Based on the information available to the EC, the pricing for this approach is similar to that of the 2014 study, with the benefit of increased rigour. The 2015 CSPV approach includes six very high-rigour sites; four for Union and two for. These sites are included as case studies to understand the value and applicability of the high rigour approach for possible use in future evaluation years when evaluation budgets increase and program impacts become more significant. The high rigour site visits also follow the best practice within the industry for large, complex custom projects. The proposed approach will enhance that used in 2014 by: Increasing the number of on-sites by 10% and the total number of projects reviewed by 80%. Expanding the number of the projects reviewed, the EC will gain a broader and deeper view of program delivery and savings estimation practices. Coupled with the NTG study, the expanded CPSV will allow a more comprehensive and fully transparent analysis of the programs leading to recommendations for program performance improvement. The results of the handful of high rigour case studies will demonstrate the value of deep project-level review, providing helpful input into the direction for future evaluation design where some extent of these deep reviews may become standard practice as is the case in other leading jurisdictions. The value of the increased post-visit analysis time in the proposed approach is difficult to quantify; however, its impact should not be underestimated. Although strong attention to site-level savings estimates does increase the accuracy of the program-level savings estimate, it does not in and of itself provide lasting, impactful recommendations that will improve the program and help increase the accuracy of the pre-evaluation savings estimates for future years. The post-site visit analysis conducted by an experienced evaluation consultant can actually add the most value. The final result will yield more useful results overall and insights that the OEB and its stakeholders can use to help define the next cycle of evaluation research questions and issues. 26 In particular, it should be noted that the EC anticipates leveraging a separate strata for Multi-Family sites once complete information for both the standard-income and low-income Multi-Family programs is available. Currently, this proposal assumes that the necessary Multi-Family sites are included in the program segments noted, however if additional sites are needed based on the complete examination of the Multi-Family programs, the budget will increase. DNV GL 11/10/16 Page 122

138 Memo to: Natural Gas Demand Side Management Evaluation Advisory Committee (EAC) From: DNV GL-Energy Date: June 6, 2016 Prep. by: Ben Jones Proposed Areas of Change for the Custom Program Net to Gross (NTG) Study This memo presents the high level changes the evaluation team anticipates for the NTG Study as a result of integrating the 2015 custom program savings verification (CPSV) effort. 1. Scope The January 2016 OEB Decision and Order specifies that the utilities are not expected to rely on pre-determined net-to-gross adjustment factors when calculating savings for custom projects. Based on the OEB s findings, evaluating the NTG adjustment factors, particularly free ridership, of custom programs on an annual basis is consistent with the OEB s Decision and ensures that industry best practice is followed.. Originally the scope of the NTG study was to develop a factor that would be used on a go forward basis (or as a pre-determined value applied in each year following the study) and that was supported by sample sizes large enough for 90/10 precision when applied to a future program year. In practical terms this means that the estimates of error used to determine the study s precision did not use the Finite Population Correction (FPC) that reduces the error estimates that apply for retrospective evaluations. FPC reduces sample sizes required, saving time and expense for retrospective evaluation. a. CHANGE: For free ridership, the January Decision and Order means that FPC errors are applicable, reducing the required sample size and free ridership is only required for the 2015 programs (as opposed to also including 2014 programs). b. NO CHANGE: We continue to recommend that Spillover for custom programs be evaluated based program participation in the 2013 and 2014 program years to determine a factor to apply to 2015 results. Effective retrospective evaluation of spillover based on the 2015 program year would occur too late for it to be used in program filings for utility incentives and lost revenue. Spillover will include inside and outside types as originally described (Overview, Key Concepts pg 4-5). Like and unlike dimensions of spillover will also both be evaluated. The additional effort required to add unlike spillover to our planned approach is a handful of additional questions asked of a portion of respondents. The program theory for unlike spillover is that these measures are due to site audits, rather than a positive experience with a program measure. Because of this distinction we recommend evaluating attribution of unlike spillover as independent of the attribution of the original program measure. 2. NO CHANGE: Run-it-Right for will be included in the evaluation, but treated as a separate program apart from the other Custom C&I programs. Results from the evaluation of Run-it-Right will likely be applied to the Union RunSmart program as well. 3. Sampling The units of analysis and stratification defined for the NTG sampling approach may require adjustment to accommodate CPSV (Task 2: Sample Design pg 8). Stratification is most efficient when the variance among strata is greater than that within strata. DNV GL 11/10/16 Page 123

139 a. Both NTG and CPSV sampling are made more efficient when stratifying by project size and oversampling the large projects. b. For NTG we identified the most relevant distinction among projects as the difference in decision making for firms considering operational improvements or maintenance versus purchasing new equipment. c. For CPSV the primary relevant division is between projects that use standardized calculation methods and projects that are more unique. With large programs that have descriptive tracking data, stratifying by measure types is also effective as long as the divisions are not made so fine that the design loses the efficiencies of stratifying by size. If the utilities can identify the projects that use standard Excel calculators (Union) or Etools (), stratifying on this basis would allow the CPSV effort to allocate resources more efficiently and get more accurate and precise results. For the combined NTG and CPSV, stratification by Utility Program Equip vs O&M Calculation type (or measure type) Size would likely result in a too finely stratified study that loses much of the efficiency that size stratification adds. CHANGE: Our recommendation is to remove the Equip vs O&M stratification level, but retain the distinction as a reporting domain within each program in the analysis. This allows us to apply the NTG at that level, but means we will not be targeting precision levels and sample sizes for Equipment and O&M measures separately. To the extent that maintenance measures are either correlated or not with Union s standard calculators and that optimization measures are either correlated or not with Union s standard calculators and s Etools, we may have some indirect control over the Equipment and O&M sample sizes with this approach. 4. Documentation and Contact Requests (Task 3.4 pg 29) The number of projects for which the evaluation requests contact information and documentation is a function of the sample size and the anticipated response rate. The utilities indicated that their experience with past CPSV efforts suggests a higher response rate than DNV GL has seen in recent studies in the US Midwest, possibly due to having utility representatives visiting sites with the evaluation engineer and smoothing the way. For NTG, utility involvement in the data collection would result in the appearance of bias. The approach we have used in the past and currently recommend for this study is: a. The utilities send advance letters or s using utility letterhead. This communication encourages companies to cooperate with the evaluator. This is the utilities only direct involvement as part of the NTG portion of the evaluation. b. The NTG portion of the evaluation is handled on the phone by a trained evaluation engineer (either an Itron or DNV GL employee). If the site is also selected to receive an on-site, the evaluation engineer will begin scheduling the visit. To reduce any bias (real or perceived), the utility will not be included for on-site visits that require on-site completion of the NTG component. c. The utility representative (project manager and/or account manager), in concert with the assigned on-site engineer, will complete the scheduling process for all other CPSV related DNV GL 11/10/16 Page 124

140 evaluation activities. Depending on the complexity of the site, the on-site engineer may work for Stantec, DNV GL, or Itron. CHANGE: While this process will likely hold for the majority of the projects in the CPSV and NTG samples, there may be some exceptions for very large or very complex projects. DNV GL is considering on-site, in-depth NTG interviews using Itron or DNV GL engineers. In these situations, it is not appropriate to have the utility representative on-site to influence the NTG responses. Having NTG and CPSV integrated into a single study does not mean that the final completed samples for each effort are the same. Ideally they overlap completely, but there are always respondents who will respond to one, but not the other. While this reduces the response rate for each component somewhat, in our experience most companies that agree to one will agree to both. In plan, the CPSV will be either the same sample as the NTG or a subset of the NTG sample. 5. Survey Design (Task 3.3, pg 28) The study originally included 3 years for the sample (2013, 2014 and 2015) with two years of spillover (2013 and 2014)and two years of free ridership (2014 and 2015).. Under the original scope, 2014 acted as an overlap year. POTENTIAL CHANGE: Because free ridership is no longer part of the 2014 scope and because unlike spillover is included, we are considering re-arranging the spillover survey guides in a way that addresses spillover up front (following the framing section), and only asks about the free ridership of the original measure when spillover is reported by the customer. This change will reduce customer burden overall. 6. POTENTIAL CHANGE: Methodology (Task 4.1 pg 33) Depending on whether the CPSV effort attempts to verify all 2015 NTG projects or only a subset, the LCNS calculation method for the free ridership score may be employed. 7. CHANGE: Communication (Task 5 pg 34) The shift to OEB management of the contract will result in changes to the deliverables section of the work plan, specifically status reports, who receives what when, and the timing of the complete methodology memo (now timed to coincide with the draft survey/interview guides). DNV GL 11/10/16 Page 125

141 Memo to: EAC Members From: DNV GL - Energy Date: July 7, 2016 Copied to: Wendy Todd Prep. by: Tammy Kuiken 2015 Verification Options for C&I Prescriptive and Market Transformation Programs This memo provides initial, high-level options for verifying the installation or scorecard metrics of the commercial and industrial (C&I) prescriptive programs and market transformation programs offered by Union and in C&I Prescriptive The Evaluation Contractor (EC) team has devised a number of options for verifying the installation of C&I prescriptive measures installed through the Union and programs. The options are outlined in the table below, ordered from lowest to highest anticipated cost. All of the options require delivery of the entire tracking database from Union and, including the customer name, contact information, measure and quantity installed, and tracked savings. A sample must be designed which should account for measure type and size, and a sample of projects must be pulled for verification. In all cases, any installation discrepancies would be noted, with partial installation credited to the program for a lower installed quantity, and no installation credited if the measure does not qualify for a program incentive. If a measure was entered in the tracking data incorrectly, the appropriate savings will be identified and the program will be credited with installation of the ratio between the tracking and verified savings (which can be greater than or less than 100%). Once all of the sampled projects have been reviewed, a weighted overall installation rate will be calculated for the program. The cost of the telephone verification depends in large part on the information available in the electronic tracking data. If the tracking data includes details such as the location where the measure was installed, the details of the measure installed (such as the size of the shipping door), and the quantity of the measure installed, the telephone verification may be a lower cost option then the desk review. However, if some of the required information is contained only in paper documentation, then this option will likely cost more. The C&I verification process can include one or all three of the options in differing combinations. For example, the desk review could be completed for the majority of measures, with telephone interviews conducted only for those with insufficient documentation and on-sites conducted when the telephone information could not confirm installation. For the C&I prescriptive program, the EC recommends telephone verification supplemented with on-site verification for large or unusual projects. It may be possible to conduct some on-site verification in the course of completing custom program verification visits. DNV GL 11/10/16 Page 126

142 Verification Options for C&I Prescriptive Option Pros to Option Cons to Option Desk Review. The verification contractor will request all electronic and paper documentation associated with each of the sampled measures. To complete the verification, the contractor will compare the submitted paperwork with the incentive application and tracking data to ensure that the information was correctly documented at all points of the process. The contractor will also compare the installation date, application date, and invoice details (if available) to ensure the project qualified for installation under program rules. Telephone Verification. If the tracking data does not have the necessary details, then the verification contractor will request the paper documentation for a sample of projects equal to the desired number of survey completes times three or four, depending on the anticipated response rate. The contractor will pull the necessary information from the paperwork and enter it into a database for use in the CATI survey. If the tracking data contains the necessary details (or once it does), then the verification contractor will write a computer-aided telephone interview (CATI) guide that can be used to verify the installation of each sampled measure. The CATI survey will use skip patterns based on the tracking data and measure-specific language to determine whether a measure was installed and to verify that it qualified for a program incentive. Least cost option. Assuming the site contact is able to truthfully verify the necessary information, confirms the equipment, installation location, and installation. Allows contractor to verify the program was implemented consistently and as documented. Assumes that the equipment represented by the submitted documentation was installed correctly at the installation location and remains installed for the equipment life. While the reliability of this method is generally good, there can be problems with uninformed site contacts. Standard practice is to assume it s installed unless the contractor can prove otherwise. Assumes the equipment is installed for the remaining life. DNV GL 11/10/16 Page 127

143 Option Pros to Option Cons to Option On-site Verification. The verification contractor will recruit and visit the installation location for a sample of measures. While on location, the contractor will view the installed equipment and verify the submitted information, including model number, size, and capacity as applicable. Most accurate. Verification contractor can visually confirm the equipment, location, and quality of installation. Allows contractor to verify the program was implemented consistently and as documented. Costs the most by far. Still assumes the equipment will be installed for the remaining life. DNV GL 11/10/16 Page 128

144 Market Transformation: Residential savings by design and Optimum Home The 2015 scorecard metrics for the Residential Savings by Design program are the number of builders enrolled (18) and the number of homes built (1,111). To qualify, an enrolled builder must complete at least 50 new homes each year (prior to participation) and have completed less than 200 homes within the program, and the home must exceed the current building code by 25%. 27 The Evaluation Contractor (EC) team has devised a number of options for verifying that the metrics have been met. The options are outlined in the table below, ordered from lowest to highest anticipated cost. All of the options require delivery of the program tracking database from and Union for all program years since the program s inception (2012), including the participant name, contact information, and number of homes built. The 2015 scorecard metric for the Union Optimum Home program is the portion of new buildings that exceed the current building code by at least 20%. 28 To meet the metric in 2015, almost 30% of the homes built in Union territory must qualify. The Evaluation Contractor (EC) team recommends verification activities similar to those outlined for the Residential Savings by Design program. In addition to those activities, the verification contractor will need to determine the total number of homes built in Union territory in 2015 in order to determine the portion that are energy efficient. The Residential Savings by Design evaluation can include one or all of the options. For example, the tracking data and telephone verification can be implemented on the entire population of builders. If desired, the on-site verification or building simulations could be conducted on a sample. The budget will not likely support on-site or building simulation on a representative sample of homes; however, completing one visit per builder or a limited number (3-5) of building simulations might indicate whether further study is necessary. For the Residential Savings by Design and Optimum Home programs, the EC recommends telephone verification. Although the on-site visits will increase the appearance of rigor, the actual benefit received will be low unless a representative sample can be drawn, which requires all of the builders records. The building simulation would confirm the actual energy savings resulting from the programs, but since energy savings are not included in program metrics, the outcome will not affect the shareholder incentive. Therefore, the telephone verification should be sufficient. 27 It s unclear what metric should be used to gauge whether the home has exceeded the building code. Some possibilities include: installed systems individually exceed building code by 25%; overall home energy use is 25% lower than the typical building in the same climate and of the same size; or a clearly defined energy rating (such as the HERS rating in the US) for the new building is 25% better than the rating for a typical building. Other definitions may also apply. 28 Again, it s unclear what metric should be used to gauge whether the home has exceeded the building code. DNV GL 11/10/16 Page 129

145 Verification Options for Residential Savings by Design and Optimum Home Option Pros to Option Cons to Option Desk Review. The verification contractor will summarize the electronic tracking data, including the total number of builders participating since program inception, the number of homes built by those builders, the number of new builders enrolled in the 2015 program year, and the number of homes built by those builders in Telephone Verification. The verification contractor will phone all of the builders who participated in the 2015 program to ask them how long they had participated, how many total homes have been built through the program since inception, and how many homes they built in If the verification contractor is not able to reach all builders by phone, they will need to expand the results of the survey to the builder population. Least cost option. Assuming the site contact is able to truthfully verify the number of homes and home locations, confirms number of homes built. Does not require contact with the end-user, who likely are not aware that their house participated in an energy efficiency program. Allows contractor to verify the program was implemented consistently and as documented. Assumes that the tracking data accurately reflects program participation and all reported homes qualify for program participation (i.e. exceed building code by 25%, in utility territory). Does not increase verification rigor over past years. Assumes that all reported homes qualify for program participation (i.e. exceed building code by 25%%, in utility territory). DNV GL 11/10/16 Page 130

146 Option Pros to Option Cons to Option On-site Verification. This task starts with the activities performed in the telephone verification. In addition to participation information, the verification contractor will ask builders to provide the address and owner name for a sample of homes built under the program and a listing of the measures installed in those homes. The contractor will recruit homes for site visits and visually verify installation of the higher efficiency measures. Building Simulation. This task starts with the activities performed in the telephone verification. In addition to participation information, the verification contractor will ask builders for blueprints and equipment specifications for a sample of homes built under the program and a listing of the measures that were installed to ensure the home exceeded building code. The verification contractor will create two models of each home using a standard building simulation program: one that meets code, and one that reflects the home as it was built. The simulations will be run to estimate annual building energy use and the difference calculated to determine whether the home achieves the 25% improvement metric. Confirms that the intended measures were actually installed, which might not be the case if the builder ran into problems with the supply timeline or available resources. Allows contractor to verify the program was implemented consistently and as documented. Confirms that the homes were built to use 25% less energy than a code-compliant building. Does not require contact with the end-user, who likely are not aware that their house participated in an energy efficiency program. Allows contractor to verify the program was implemented consistently and as documented. Assumes that the installed measures result in a home that qualifies for program participation (i.e. exceeds building code by 25%) and all reported homes were built (cannot verify all of them; cannot sample without builder records). Depending on number and detail, this option could be significantly more expensive than the previous options. Requires contact with end-user. Assumes that the measures were installed as reported and all of the reported homes were built. Depending on number of homes, level of detail from builders, and level of detail of model (such as calibration with current billing records, if available), this option could be significantly more expensive than the previous options. DNV GL 11/10/16 Page 131

147 Market Transformation: Commercial Savings by Design The 2015 scorecard metric for the Commercial Savings by Design programs is the number of new developments enrolled. To qualify, a development must be at least 50,000 square feet in size (through one or multiple buildings of the same design), be designed using an integrated design process (IDP), and be at least 25% better than current building code. The Evaluation Contractor (EC) team has devised a number of options for verifying that the metric has been met. The options are outlined in the table below, ordered from lowest to highest anticipated cost. All of the options require delivery of the program tracking database for all program years since the program s inception, including the participant name, contact information, and number of buildings built. The Commercial Savings by Design evaluation can include one or all of the options. For example, the tracking data and telephone verification can be implemented on the entire population of builders. If desired, the on-site verification or building simulations could be conducted on a sample. The budget will not likely support on-site or building simulation on a representative sample of buildings; however, completing one visit per builder or a limited number (3-5) of building simulations might indicate whether further study is necessary. For the Commercial Savings by Design program, the EC recommends telephone verification. Although the on-site visits will increase the appearance of rigor, the actual benefit received will be low unless a representative sample can be drawn, which requires all of the builders records. The building simulation would confirm the actual energy savings resulting from the programs, but since energy savings are not included in program metrics, the outcome will not affect the shareholder incentive. Therefore, the telephone verification should be sufficient. DNV GL 11/10/16 Page 132

148 Verification Options for Commercial Savings by Design Option Pros to Option Cons to Option Desk Review. The verification contractor will summarize the electronic tracking data, including the total number of builders participating in the program, the number of developments initiated by those builders, and the number completed in Telephone Verification. The verification contractor will phone all of the builders who participated in the 2015 program to ask them how long they had participated, how many total developments have been built through the program since inception, and how many developments were in progress or finished in If the verification contractor is not able to reach all builders by phone, they will need to expand the results of the survey to the builder population. Least cost option. Assuming the site contact is able to truthfully verify the number of developments, confirms number of developments built/started. Does not require contact with the end-user, who likely are not aware that their development participated in an energy efficiency program. Allows contractor to verify the program was implemented consistently and as documented. Assumes that the tracking data accurately reflects program participation and all reported buildings qualify for program participation (i.e. exceed building code by 25%, in utility territory). Does not increase verification rigor over past years. Assumes that all reported developments qualify for program participation (i.e. exceed building code by 25%%, in utility territory). DNV GL 11/10/16 Page 133

149 Option Pros to Option Cons to Option On-site Verification. This task starts with the activities performed in the telephone verification. In addition to participation information, the verification contractor will ask builders to provide the address and owner name for a sample of developments built under the program and a listing of the measures installed in those developments. The contractor will recruit developments for site visits and visually verify installation of the higher efficiency measures. Building Simulation. This task starts with the activities performed in the telephone verification. In addition to participation information, the verification contractor will ask builders for blueprints and equipment specifications for a sample of developments built under the program and a listing of the measures that were installed to ensure the building exceeded building code. The verification contractor will create two models of each building using a standard building simulation program: one that meets code, and one that reflects the building as it was built. The simulations will be run to estimate annual building energy use and the difference calculated to determine whether the building achieves the 25% improvement metric. Confirms that the intended measures were actually installed, which might not be the case if the builder ran into problems with the supply timeline or available resources. Allows contractor to verify the program was implemented consistently and as documented. Confirms that the developments were built to use 25% less energy than a code-compliant building. Does not require contact with the end-user, who likely are not aware that their building participated in an energy efficiency program. Assumes that the installed measures result in a building that qualifies for program participation (i.e. exceeds building code by 25%) and all reported developments were built (cannot verify all of them; cannot sample without builder records). Depending on number and detail, this option could be significantly more expensive than the previous options. Requires contact with end-user. Assumes that the measures were installed as reported and all of the reported developments were built. Depending on number, level of detail from builders, and level of detail of model (such as calibration with current billing records, if available), this option could be significantly more expensive than the previous options. DNV GL 11/10/16 Page 134

150 APPENDIX C. HISTORIC SAVINGS AND BUDGET This appendix contains tables that show the annual and cumulative energy savings, budget, and savings per budget for each program, where available. Historic savings and budget (annual savings): Home Energy Conservation and Home Reno Rebate Program Year Savings Claimed or Achieved, m 3 Budget Spent, $ m 3 /$ Union Gas 2015 (Claimed) 3,254,128 $2,202, (Achieved) 1,342,361 $1,997, (Achieved) 303,672 $1,181, (Achieved) 89,969 $178, (Claimed) 6,762,791 $9,362, (Achieved) 5,914,881 $8,605, (Achieved) 1,949,026 $2,376, (Achieved) n/a n/a n/a Source: From 2012, 2013, 2014 and 2015 (draft) DSM Annual Reports. Union Budget Spent: EB , Exhibit B.T5.Union.CCC.26. Actual spending for , proposed budget for Administration and evaluation overheads are excluded budget excludes overheads. only reported cumulative savings in its 2012 annual report. Historic Savings and Budget (cumulative savings): Home Energy Conservation and Home Reno Rebate Program Year Savings Claimed or Achieved, CCM Budget Spent, $ CCM/$ Union Gas 2015 (Claimed) 58,923,165 $2,202, (Achieved) 26,518,351 $1,997, (Achieved) 6,073,437 $1,181, (Achieved) 1,799,370 $178, (Claimed) 102,415,214 $9,362, DNV GL 11/10/16 Page 135

151 Program Year Savings Claimed or Achieved, CCM Budget Spent, $ CCM/$ 2014 (Achieved) 89,690,562 $8,605, (Achieved) 38,980,521 $2,376, (Achieved) 5,296,300 $816, Source: From 2012, 2013, 2014 and 2015 (draft) DSM Annual Reports. Union Budget Spent: EB , Exhibit B.T5.Union.CCC.26. Actual spending for , proposed budget for Administration and evaluation overheads are excluded budget excludes overheads. Historic savings and budget (annual savings): Home Winterproofing, Home Weatherization, Furnace End-of-life, and Aboriginal Offerings Program Year Savings Claimed or Achieved, m 3 Budget Spent, $ m 3 /$ Union Gas 2015 (Claimed) 1,341,913 $4,442, (Achieved) 1,446,863 $5,239, (Achieved) 1,618,601 $5,488, (Achieved) 1,491,631 $5,242, (Claimed) 1,139,959 $4,444, (Achieved) 1,036,919 $4,494, (Achieved) 1,375,598 $4,639, (Achieved) n/a n/a n/a Source: From 2012, 2013, 2014 and 2015 (draft) DSM Annual Reports. EB , Exhibit B.T3.VECC.28 Union, actual spending , proposed budget Administration and evaluation overheads are excluded budget excludes overheads. Historic savings and budget (cumulative savings): Home Winterproofing, Home Weatherization, Furnace End-of-life, and Aboriginal Offerings Program Year Savings Claimed or Achieved, CCM Budget Spent, $ CCM/$ Union Gas 2015 (Claimed) 33,504,841 $4,442, (Achieved) 36,105,327 $5,239, DNV GL 11/10/16 Page 136

152 Program Year Savings Claimed or Achieved, CCM Budget Spent, $ CCM/$ 2013 (Achieved) 40,236,650 $5,488, (Achieved) 36,342,693 $5,242, (Claimed) 28,343,978 $4,444, (Achieved) 25,673,482 $4,494, (Achieved) 32,904,684 $4,639, (Achieved) 24,708,220 $5,758, Source: From 2012, 2013, 2014 and 2015 (draft) DSM Annual Reports. EB , Exhibit B.T3.VECC.28 Union, actual spending , proposed budget Administration and evaluation overheads are excluded budget excludes overheads. Historic savings and budget (annual savings): Low Income Multi Residential Affordable Housing and Low-income Multi-family Offering Program Year Savings Claimed or Achieved, m 3 Budget Spent, $ m 3 /$ Union Gas 2015 (Claimed) 909,877 $1,385, (Achieved) 1,277,353 $2,221, (Achieved) 933,333 $1,566, (Achieved) 601,126 $1,377, (Claimed) 3,397,959 $2,111, (Achieved) 1,708,673 $1,930, (Achieved) 1,548,054 $723, (Achieved) n/a n/a n/a Source: From 2012, 2013, 2014 and 2015 (draft) DSM Annual Reports. Union Budget spent : EB , Exhibit B.T3.Union.FRPO.1., EB , Exhibit B.T3.VECC.28. Administration and evaluation overheads are excluded budget excludes overheads. DNV GL 11/10/16 Page 137

153 Historic savings and budget (cumulative savings): Low Income Multi Residential Affordable Housing and Low-income Multi-family Offering Program Year Savings Claimed or Achieved, CCM Budget Spent, $ CCM/$ Union Gas 2015 (Claimed) 16,965,778 $1,385, (Achieved) 21,586,843 $2,221, (Achieved) 15,267,883 $1,566, (Achieved) 11,871,819 $1,377, (Claimed) 69,226,782 $2,111, (Achieved) 29,801,158 $1,930, (Achieved) 27,314,154 $723, (Achieved) 43,407,789 $1,367, Source: From 2012, 2013, 2014 and 2015 (draft) DSM Annual Reports. Union Budget spent : EB , Exhibit B.T3.Union.FRPO.1., EB , Exhibit B.T3.VECC.28. Administration and evaluation overheads are excluded budget excludes overheads. Historic savings and budget (cumulative savings): Large Volume Program Program Year Savings Claimed or Achieved, CCM Budget Spent, $ CCM/$ Union Gas 2015 (Claimed) 578,023,195 $3,209, (Achieved) 870,195,452 $4,101, Source: Schedule B Decision and order OEB-approved scorecard weighting Gas Distribution, Inc. (EB ) / Union Gas Limited (EB ) January 20, 2016 Historic savings and budget (cumulative savings): Commercial and Industrial Prescriptive Energy Solutions and Prescriptive and Quasi-prescriptive Offering Program Year Savings Claimed or Achieved, CCM Budget Spent, $ CCM/$ Union Gas 2015 (Claimed) 183,095, (Achieved) 216,057,244 $4,957, DNV GL 11/10/16 Page 138

154 2015 (Claimed) 106,286, (Achieved) 86,666,513 $767, Source: Schedule B Decision and order OEB-approved scorecard weighting Gas Distribution, Inc. (EB ) / Union Gas Limited (EB ) January 20, budget values were not broken out for C&I Prescriptive and Custom projects. Historic savings and budget (cumulative savings): Commercial and Industrial Custom Energy Solutions and C&I Custom Program Year Savings Claimed or Achieved, CCM Budget Spent, $ CCM/$ Union Gas 2015 (Claimed) 678,002, (Achieved) 683,855,047 $7,784, (Claimed) 556,659, (Achieved) 484,885,481 $5,713, Source: Schedule B Decision and order OEB-approved scorecard weighting Gas Distribution, Inc. (EB ) / Union Gas Limited (EB ) January 20, budget values were not broken out for C&I Prescriptive and Custom projects DNV GL 11/10/16 Page 139

155 APPENDIX D. LOGIC MODELS This appendix contains program logic models provided by Union Gas, where available. The logic models show, at a very high level, how the services and inputs offered by the program result in the desired outcomes. Although these logic models are specific to Union, we assume the logic models are similar where similar programs are offered. Union Home Reno Rebate logic model DNV GL 11/10/16 Page 140

156 Union Home Weatherization logic model DNV GL 11/10/16 Page 141

157 Union Aboriginal Offering logic model DNV GL 11/10/16 Page 142

158 Union Low Income Multi-Family Offering logic model DNV GL 11/10/16 Page 143

159 Union Commercial and Industrial Prescriptive and Quasi-prescriptive Offering logic model DNV GL 11/10/16 Page 144

160 Union Commercial and Industrial Custom Offering logic model DNV GL 11/10/16 Page 145

161 Union Optimum Home logic model DNV GL 11/10/16 Page 146

162 Union Run Smart logic model DNV GL 11/10/16 Page 147

163 Union Strategic Energy Management logic model DNV GL 11/10/16 Page 148

164 APPENDIX E. Customer Incentives and Other Services This appendix contains a description of the incentives and other services offered by the energy efficiency programs in each portfolio. Home Energy Conservation () Home Reno Rebate (Union) 29 The programs offer very different incentive structures to participants. s are based on modeled gas savings, while Union s are prescriptive. Both utilities cover some of the costs of home energy evaluations. s incentives are based on modelled natural gas savings as a result of measures installed. Beginning in 2016, is introducing the following tiered incentive structure: 30 Up to $500 for full (pre and post) home energy evaluations, not including HST $1,100 for reaching 26%-49% annual gas savings (incremental to the $500 for home energy evaluation) $1,600 for reaching 50% and above in annual gas savings (incremental to the $500 for home energy evaluation). Union provides prescriptive incentives to customers for each installed measure, as shown in the next table. Measures, incentives, and savings: Home Energy Conservation and Home Reno Rebate Measure Description Savings per Measure Incentive Amount Union Gas Home energy evaluations Not applicable $500 max. Adding at least R23 to 100% of basement Based on HOT2000 modeling $1000 Adding at least R12 to 100% of basement Based on HOT2000 modeling $500 Adding at least R23 to 100% of crawl space wall Adding at least R10 to 100% of crawl space wall Adding at least R24 to 100% of floor above crawl space Wall: Add at least R9 for 100% of building to achieve a minimum of R12 Wall: Add at least R3.8 for 100% of building to achieve a minimum of R12 Based on HOT2000 modeling $800 Based on HOT2000 modeling $400 Based on HOT2000 modeling $450 Based on HOT2000 modeling $1,500 Based on HOT2000 modeling $1, Sources : s ( and Union s ( websites (as of ). 30 Program documentation refers to a $500 incentive for savings ranging from 10% to 25%, but this does not appear on the website (where incentives are offered starting at 25% savings). DNV GL 11/10/16 Page 149

165 Increasing attic insulation to at least R50 from R12 or less Increasing attic insulation to at least R50 from R13 to R25 Increasing cathedral/flat roof insulation by at least R14 Air sealing: Achieve 10% or more above base target Based on HOT2000 modeling $500 Based on HOT2000 modeling $250 Based on HOT2000 modeling $500 Based on HOT2000 modeling $150 Air sealing: Achieve base target Based on HOT2000 modeling $100 Replacing 94% AFUE or less furnace with 95% or higher condensing natural gas furnace or replacing 89% AFUE or less boiler with a 90% AFUE or higher ENERGY STAR condensing gas boiler Replacing a natural gas water heater with EF lower than 0.82 with an ENERGY STAR natural gas water heater with EF of 0.82 or higher For each window, door or skylight replaced with ENERGY STAR-qualified model Based on HOT2000 modeling $500 Based on HOT2000 modeling $200 Based on HOT2000 modeling $40 Home energy evaluations Not applicable $500 max. Custom measures Based on HOT2000 modeling Based on HOT2000 modeling and tiered performance-based incentive structure (see incentive description above) Both Union s and s savings are based on whole-house energy modeling. Average annual gas savings per house were 1,198 m 3 for and 986 m 3 for Union in Starting 2016, Union uses minimal baseline AFUE values for furnaces and boilers. Residential Adaptive Thermostats () A rebate of 100$ is offered to customers 31 upon proof of purchase and qualified / approved installation. 31 From s website ( consulted Program documentation refers to a $75 incentive. DNV GL 11/10/16 Page 150

166 Measures, incentives, and savings: Residential Adaptive Thermostats Measure Description Savings per Measure Incentive Amount Residential Adaptive Thermostat rebate m 3 / year $100 bill credit on account Home Winterproofing () Home Weatherization, Furnace End-of-Life, and Aboriginal Offerings (Union) The Winterproofing and Weatherization offerings provide a free home assessment and weatherization services (i.e., insulation and air sealing) to qualified customers who meet income and customer eligibility criteria. As direct install offers, there is no financial cost to the participant for the energy assessment or for the weatherization products and services. As a health and safety value-add on, a carbon monoxide monitor is provided where one is not present in the home. At the time of assessment, the home is also prequalified for basic measures. The Union Furnace End-of-Life Incentive offering provides Social and Assisted Housing Providers with an incentive amount that is equal to half of the incremental cost of upgrading to a 95% or greater efficiency rating (AFUE) furnace. Only half of the incremental costs will be covered within this market to reflect that Social and Assisted Housing Providers can gain access to additional funds, whereas private market customers do not have this option. Private market customers, including those on Aboriginal reserves, will be provided with an incentive equal to the incremental cost of upgrading to a 95% or greater efficiency rating (AFUE) furnace. Measures, incentives, and savings: Home Winterproofing, Home Weatherization, Furnace End-of- Life, and Aboriginal Offerings Measure Description Savings per Measure Incentive Amount Union Gas Home assessment, weatherization, water conservation, programmable thermostat, carbon monoxide monitor Furnace replacement Based on HOT2000 modeling. Average 2015 annual savings of 912 m 3 per home. 781 m 3 for 60% AFUE baseline 466 m 3 for 70% AFUE baseline All measures free of charge 100% of incremental cost (50% of incremental cost for Social and Assisted Housing Providers) DNV GL 11/10/16 Page 151

167 Measure Description Savings per Measure Incentive Amount Home assessment, weatherization, water conservation, programmable thermostat, carbon monoxide monitor Based on HOT2000 modeling. Average 2015 annual savings of 849 m 3 per home. All measures free of charge also has the intention of adding a furnace replacement component in We found no evidence yet of such an offering on s website or program brochure. Low Income Multi Residential Affordable Housing () Low Income Multi-Family Offering (Union) offers: Half the cost of an energy audit, up to $5,000 per building or $0.01/m3 of gas consumed in the past calendar year (whichever is less) Fixed incentives for smaller types of equipment (condensing boilers, high efficiency boilers, energy recovery ventilation systems; and heat recovery ventilation systems) In-suite direct install (free) measures (showerheads, heat reflector panels) Custom incentives based on projected first year natural gas savings at a rate of $0.40/m 3 saved ($0.20/m 3 for operational improvements), up to 50% of the cost of the retrofit Union offers: Free, comprehensive energy audits (up to $5,000/site for up to five sites) All of the prescriptive measures offered to the multi-family segment within the standard Commercial portfolio, but at a higher incentive level Custom incentives for cost effective measures: $0.10/m 3 of lifetime savings, up to 50% of eligible project costs Measures, incentives, and savings: Low Income Multi Residential Affordable Housing and Low Income Multi-Family Offering Measure Description Savings per Measure Incentive Amount Union Gas Free, comprehensive energy audit Not applicable Up to $5,000 per site Condensing Boiler (up to 299 MBH) m 3 /Btu/hr 50% of installed cost, up to $5,000 Condensing Boiler (300+ MBH) m 3 /Btu/hr 50% of installed cost, up to $11,000 Energy Recovery Ventilators (all units are eligible) 5.77 m 3 /CFM (new) 6.12 m 3 /CFM (retrofit) 50% of installed cost, up to $85,000 DNV GL 11/10/16 Page 152

168 Measure Description Savings per Measure Incentive Amount Heat Recovery Ventilators (all units are eligible) 4.28 m 3 /CFM (new) 4.70 m 3 /CFM (retrofit) 50% of installed cost, up to $125,000 Condensing Make-Up Air Units 0.84 to 2.92 m 3 /CFM 50% of installed cost, up to $38,200 Showerheads, bathroom aerators and kitchen aerators 3.73 to 20 m 3 for aerators, 7.6 to 69 m 3 for showerheads Free items and installation Custom incentive Custom assessment $0.10/m 3 of lifetime savings, up to 50% of eligible project costs Energy Audit Not applicable Half of the cost of audit, up to $5,000 or $0.01 per m 3 of gas consumed Condensing Boiler Up to 299 MBH, AFUE of 90%+ Condensing Make-Up Air Unit (single speed, up to 14,000 CFM) Condensing Make-Up Air Unit (two speed, up to 14,000 CFM) High-Efficiency Seasonal Boiler ( MBH) High-Efficiency Seasonal Boiler ( MBH) High-Efficiency Seasonal Boiler ( MBH) m 3 /Btu/hr $1, m 3 /CFM Greater of $1,000 or $0.50/CFM 2.92 m 3 /CFM Greater of $1,000 or $1.00/CFM m 3 /Btu/hr $1, m 3 /Btu/hr $2, m 3 /Btu/hr $3,000 High-Efficiency Seasonal Boiler ( MBH) High-Efficiency Non Seasonal Boiler ( MBH) High-Efficiency Non Seasonal Boiler ( MBH) m 3 /Btu/hr $4, m 3 /Btu/hr $1, m 3 /Btu/hr $2,000 High-Efficiency Non Seasonal m 3 /Btu/hr $2,500 DNV GL 11/10/16 Page 153

169 Measure Description Savings per Measure Incentive Amount Boiler ( MBH) Energy Recovery Ventilation (Minimum Heat Recovery Effectiveness 67%) Heat Recovery Ventilation (Minimum Heat Recovery Effectiveness 61%) 5.77 m 3 /CFM (new) 6.12 m 3 /CFM (retrofit) 4.28 m 3 /CFM (new) 4.70 m 3 /CFM (retrofit) $2.00/CFM $2.00/CFM Custom Incentive Custom assessment 40 cents/m 3, up to 50% of the cost of retrofit In-suite measures (showerheads, heat reflector panels) 143 m 3 for heat reflector panels,* 7.6 to 69 m 3 for showerheads Free of charge Savings from Commercial sector when not available in the Low Income sector measure assumptions list. Incentive for gas customers. In addition, Union provides a $100 incentive for most items to service providers. *From low income residential (not multi) measure assumption list. Low Income New Construction () The Low Income New Construction offer has been informed by s Savings by Design offers. However, due to the wide range of builders, types and project sizes, certain offer elements have been modified to meet the needs of the target market. Some offer elements, such as the use of the integrated design process inclusive of charrettes and incentives per unit built to energy efficiency standards, will be included in the Low Income New Construction offer. The offer will provide financial incentives and extend technical support to assist affordable housing builders in exceeding Ontario Building Code requirements by at least 15%. The offer will engage the affordable housing community to encourage adoption of energy efficiency measures and technologies as a means to maintain housing affordability. Eligibility for the offer will be simplified to encourage broader participation among large and small affordable housing participants. Specifically, the offer contemplates the following: -funded workshops, modelling tools, and charrettes that promote the integrated design process, and energy efficient and green construction practices Access to an energy efficiency/sustainability consultant during the various phases of project design Energy education materials targeted to future residents as part of their welcome to your new home package For Part 3 (Multi-Residential) developments: Tiered financial incentives depending on the modelled savings achievement. For example, 15%, 20%, 25% above code costs will be $3,000, $10,000; and $25,000 respectively, plus an additional incentive up to $2,250 per unit, to a maximum of $100,000 per building At building completion, provide a commissioning incentive of up to $10,000 DNV GL 11/10/16 Page 154

170 For Part 9 (Residential) developments that build an Energy Star home : A prescriptive incentive of up to $3,000 will be provided per home The financial incentives are indicative of s intended approach, but are subject to change as informed by the 2015 pilot. It is important to note that in 2017 there will be a change to the Ontario Building Code. The expected amendment is anticipated to include an increase to the required energy efficiency of newly constructed buildings of approximately 15% above the Ontario Building Code of As such, all programs that seek to address the overall efficiency of a building will need to be addressed as the updated Code comes into effect. Measures, incentives, and savings: Low Income New Construction Measure Description Savings per Measure Incentive Amount Part 3 Developments Custom Tiered incentives based on modeled savings (max. $100,000 per building) Part 3 Commissioning Not available $10,000 Part 9 Developments Not available $3,000 per home Large Volume (Union) The program components are listed below. Customer Engagement Communication and Education. Provide education, training and technical expertise and offer a wide variety of materials aimed at building an increased awareness of energy efficiency opportunities and benefits. New Equipment and Processes. With the continual focus on cost reduction, industrial facilities lack the resources required to analyze potential energy saving opportunities. Union Large Volume offers incentives designed to influence equipment choices to help companies control energy costs. Operations and Maintenance. Projects in the Union Large Volume program are related to the repair, replacement, or optimization of an existing piece of equipment or system. Process Improvement Studies. Union s Large Volume program works to support performance testing and analyses of industrial boilers, total steam plants, thermal fluid heaters, vaporizers, furnaces and special process equipment. Engineering Feasibility Studies. Engineering feasibility studies include an analysis of natural gas equipment as well as electricity, compressed air, water and wastewater. Steam Trap Surveys. Steam trap surveys conducted by qualified service companies are designed to identify losses from steam distribution systems. Boiler Tune ups. Union provides incentives to large volume industrial customers for the optimization of their facilities boiler air to fuel ratio DNV GL 11/10/16 Page 155

171 2015 Measures, incentives, and savings: Large Volume Measure Description Savings per Measure Incentive Amount Union Gas: Rate T1 Engineering feasibility study Custom calculation 50% of study cost up to $10,000 Process improvement study Custom calculation 66% of study cost, up to $20,000 Steam trap survey Custom calculation 50% of cost up to $6,000 New equipment Custom calculation $0.10 per CCM, up to $40,000 Operations and maintenance Custom calculation $0.10 per CCM, up to $20,000 Boiler tune-up Custom calculation $250 per boiler Meters gas/steam/hot water Custom calculation 50% of cost up to $1,000 per meter Infrared polyethylene Custom calculation $100 per growing acre Union Gas: Rate T2/Rate 100 Engineering feasibility study Custom calculation 50% of study cost up to $10,000 Process improvement study Custom calculation 66% of study cost, up to $20,000 Steam trap survey Custom calculation 50% of cost up to $6,000 Direct access budget new equipment Aggregate pool funded new equipment Direct access budget operations and maintenance Custom calculation $0.08 per CCM, up to $40,000 Custom calculation $0.05 per CCM, up to $20,000 Custom calculation $0.08 per CCM, up to $20,000 Aggregate pool funded operations and maintenance Custom calculation $0.05 per CCM, up to $10,000 Meters gas/steam/hot water Custom calculation 50% of cost up to $1,000 per meter also has the intention of adding a furnace replacement component in We found no evidence yet of such an offering on s website or program brochure. DNV GL 11/10/16 Page 156

172 Commercial and Industrial Prescriptive Energy Solutions () Prescriptive and Quasi-Prescriptive Offering (Union) The program offerings are shown in the following table Measures, incentives, and savings: Commercial and Industrial Prescriptive Energy Solutions and Prescriptive and Quasi-Prescriptive Offering Measure Description Savings per Measure Incentive Amount Union Gas Cooking equipment: ENERGY STAR fryer Demand control ventilation kitchen fast food up to 4,999 CFM Demand control ventilation kitchen full menu 5,000 to 9,999 CFM Demand control ventilation dinner house 10,000 to 15,000 CFM Demand control ventilation (DCV) retail, rooftop unit (RTU) or make-up air (MUA) less than 5,000 ft 2 DCV retail, RTU/MUA greater than or equal to 5,000 ft 2 1,408 m 3 /year $50-$200 4,801 m 3 /year $100-$1,200 11,486 m 3 /year $100-$3,000 18,924 m 3 /year m 3 /year/ft 2 $50-$ m 3 /year/ft 2 $50-$350 DCV office, RTU/MUA less m 3 /year/ft 2 $50-$100 than 2,500 ft 2 DCV office, RTU/MUA greater m 3 /year/ft 2 $50-$200 than or equal to 2,500 ft 2 Infrared heating 0 1, m 3 /year/kbtuh $50-$300 Heat recovery ventilator (HRV) multi-family, health care, nursing HRV Office, warehouse, school 4.28 m 3 /CFM (new) 4.70 m 3 /CFM (retrofit) 1.52 m 3 /CFM (new) 1.67 m 3 /CFM (retrofit) $50-$400 $50-$400 DNV GL 11/10/16 Page 157

173 Measure Description Savings per Measure Incentive Amount HRV hotel, restaurant, retail Energy recovery ventilator (ERV) multi-family, health care, nursing ERV Office, warehouse, school ERV hotel, restaurant, retail 2.38 m 3 /CFM (new) 2.61 m 3 /CFM (retrofit) 5.77 m 3 /CFM (new) 6.12 m 3 /CFM (retrofit) 2.05 m 3 /CFM (new) 2.17 m 3 /CFM (retrofit) 3.21 m 3 /CFM (new) 3.4 m 3 /CFM (retrofit) $50-$700 $50-$600 $50-$600 $50-$600 Destratification fan 0.5 m 3 /square foot $100-$1,300 Condensing rooftop units (MUA) MR and LTC Condensing rooftop units (MUA) Retail and Commercial Buildings Condensing boiler under 300 MBH m 3 /year/cfm $100-$ m 3 /year/cfm $100-$1, m 3 /year/btuh $50-$600 Condensing boiler 300 to 999 MBH Condensing boiler 1,000+ MBH Air curtains: 48 to 96 ft 2, pedestrian Air curtains: 96 ft 2 +, pedestrian Air curtains: 64 to 80 ft 2, shipping and receiving m 3 /year/btuh $50-$1, m 3 /year/btuh $50-$4,500 1,529 m 3 /year $100-$250 1,529 m 3 /year $100-$500 7,565 m 3 /year $100-$1,000 Air curtains: 80 to 100 ft 2 9,457 m 3 /year $100-$1,000 Air curtains: 100 ft ,605 m 3 /year $100-$1,500 Condensing water heater, 332 1,551m 3 /year $50-$350 DNV GL 11/10/16 Page 158

174 Measure Description Savings per Measure Incentive Amount 100, 500, 1,000 gal/day/tank Laundry washing equipment with Ozone, up to 120 pounds and 100,000 to 199,999 pounds per year Laundry washing equipment with Ozone, up to 120 pounds and 200,000+ pounds per year Laundry washing equipment with Ozone, 120+ pounds and 260,000+ pounds per year Dishwasher: stationary rack and under counter, ENERGY STAR Dishwasher: rack conveyor single and multi-tank, ENERGY STAR m 3 /year/lb/yr $100-$1, m 3 /year/lb/yr $100-$1, m 3 /year/lb/yr $100-$6, ,120 m 3 /year $50-$ ,469 m 3 /year $50-$400 Air door: single door (pedestrian entrance) Air door: double door (pedestrian entrance) Air door: shipping doors (8 x8 ) Air door: shipping doors (8 x10 ) Air door: shipping doors (10 x10 ) Condensing boilers: up to 299 MBH, AFUE of 90% or 765 m 3 /year $200 1,529 m 3 /year $300 7,565 m 3 /year $900 9,457 m 3 /year $1,200 20,605 m 3 /year $2, m 3 /year/btuh $600 DNV GL 11/10/16 Page 159

175 greater Measure Description Savings per Measure Incentive Amount Condensing make-up air (MUA) units: single speed up to 14,000 CFM, multi-family and long-term care Condensing (MUA) units: two speed up to 14,000 CFM, multi-family and long-term care Condensing (MUA) units: single speed up to 14,000 CFM, other sectors Condensing (MUA) units: two speed up to 14,000 CFM, other sectors Demand control ventilation: single zone retail Demand control ventilation: single zone offices m 3 /year/cfm Greater of $500 or $0.15/CFM m 3 /year/cfm Greater of $500 or $0.60/CFM m 3 /year/cfm Greater of $500 or $0.15/CFM m 3 /year/cfm Greater of $500 or $0.30/CFM m 3 /year/ft 2 $0.04/ft m 3 /year/ft 2 $0.04/ft 2 Destratification fans: minimum diameter of 20, minimum of 25 ceilings Energy recovery ventilation (ERV): offices, warehouses, and schools; minimum heat recovery effectiveness 67% Energy recovery ventilation (ERV): hotels, restaurants, and retail; minimum heat recovery effectiveness 67% Energy recovery ventilation (ERV): multi-family, healthcare, and long term care; minimum heat recovery 0.5 m 3 /cubic foot/year $ m 3 /year/cfm $0.75/CFM m 3 /year/cfm $0.80/CFM m 3 /year/cfm $1.50/CFM DNV GL 11/10/16 Page 160

176 Measure Description Savings per Measure Incentive Amount effectiveness 67% Heat recovery ventilation (HRV): offices, warehouses, and schools; minimum heat recovery effectiveness 61% Heat recovery ventilation (HRV): hotels, restaurants, and retail; minimum heat recovery effectiveness 61% Heat recovery ventilation (HRV): multi-family, healthcare, and long term care; minimum heat recovery effectiveness 61% High efficiency boilers up to 599 MBH High efficiency boilers up to 600 to 999 MBH High efficiency boilers up to 1,000 to 1,500 MBH m 3 /year/cfm $0.40/CFM m 3 /year/cfm $0.60/CFM m 3 /year/cfm $1.10/CFM m 3 /year/btuh $600 2,474-27,325 m 3 /year $850 2,474-27,325 m 3 /year $1,400 High efficiency boilers up to 1,501 to 2,000 MBH High efficiency boilers for school boards: elementary schools High efficiency boilers for school boards: secondary schools Infrared heaters: single stage up to 300,000 Btu Infrared heaters: two stage up to 300,000 Btu 2,474-27,325 m 3 /year $2,200 12,217 m 3 /year $1,000 49,476 m 3 /year $4, m 3 /year/kbtuh $ m 3 /year/kbtuh $100 Ozone laundry system: m 3 /year/lb/yr $0.01 per annual pound of DNV GL 11/10/16 Page 161

177 Measure Description Savings per Measure Incentive Amount commercial laundry washing Showerhead: multi-family residential buildings replacing 2.5 GPM or greater with 1.5 GPM or less Demand control ventilation: kitchen up to 4,999 CFM (approved manufacturers only) Demand control ventilation: kitchen up to 5,000 to 9,999 CFM (approved manufacturers only) Demand control ventilation: kitchen 10,000 to 15,000 CFM (approved manufacturers only) Dishwashers: multiple tank conveyor, ENERGY STAR qualified Dishwashers: single tank conveyor, ENERGY STAR qualified Dishwashers: under counter high temperature, ENERGY STAR qualified Dishwashers: stationary rack low and high temperature, ENERGY STAR qualified Convection oven: full size ENERGY STAR qualified Steam cookers: ENERGY STAR qualified Under-fired boilers: maximum idle energy rate: 65,000 Btu/hr; minimum laundry processed m 3 /year $6 4,801 m 3 /year $1,500 11,486 m 3 /year $3,000 18,924 m 3 /year $5,000 2,124 2,469 m 3 /year $ ,712 m 3 /year $ m 3 /year $ ,120 m 3 /year $ m 3 /year $200 8,889 m 3 /year $200 2,511-5,021 m 3 /year $200 DNV GL 11/10/16 Page 162

178 Measure Description Savings per Measure Incentive Amount efficiency 34% Source: Table 5.10 of UNION_draft DSM Annual Report_ Savings source: EB Exhibit B, Tab 1, Schedule 1 Commercial and Industrial Direct Install For the C&I Direct Install program, covers 50% of the cost of equipment and installation for air doors, infrared heaters, and demand control kitchen ventilation. will cover 100% of the cost of pre-rinse spray valves. It is assumed that the Union program offers similar measures and incentives. Commercial and Industrial Custom Energy Solutions () Custom Offering (Union) offers incentives ranging from $0.10/m 3 to $0.30/m 3 up to 50% of the project cost, not to exceed $100, Incentives are calculated based on the projected first year natural gas savings and remitted upon project completion. Eligible measures include improving boiler controls, installing primary and secondary piping to improve the efficiency of boiler controls, and converting to low-flow showerheads to conserve hot water. Union offers incentives of $0.10/m 3 of natural gas saved. 33 Small Commercial New Construction Program ( Pilot) The Small Commercial New Construction program offers the following services: 34 Technical assistance: installation of specific measures as required Training/education: Energy Modelling plan training Marketing/communication: promotion directly to builders, energy modellers, architects/engineers, contractors, communication. The customer incentives are: 35 Subsidized energy models: approximately $10,000 Performance incentives of $1.00/m 3 based on post construction modeled results. Residential Savings by Design () Optimum Home (Union) s Residential Savings by Design program provides builders and designers of new, Part 9 residential low-rise houses with support during the building design process, including: A Visioning Session leveraged to identify the builder s sustainability goals and options 32 Source: Retrieved on 6/2/2016 from: 33 Source: Section of UNION_draft DSM Annual Report_ EB Exhibit B, Tab 2, Schedule 1, Page 7 of EB Exhibit B, Tab 2, Schedule 1, Page 33 of 100 DNV GL 11/10/16 Page 163

179 An Integrated Design Process (IDP) Session to identify and evaluate strategies to meet the builder s sustainability goals and build new homes that exceed the energy code Building Energy Modelling used to evaluate each project s baselines and proposed improvements Union s Optimum Home program provides builders with consulting services from building science experts in the first phase of participation. In Phase 2, builders construct a prototype home using the learnings from Phase 1. In Phase 3, they incorporate the learned building practices to implement the Optimum Home process in other construction. Union also works with builders to identify efficiencies in their building practices and encourages them to apply the resulting cost savings to buy down the incremental cost of high efficiency homes. Finally, Union offers builders ongoing consulting support to address new technical and quality issues and education on selling higher efficiency homes. As a market transformation program, there are no specific savings associated with the program. Measures, incentives, and savings: Residential Savings by Design and Optimum Home Measure Description Savings per Measure Incentive Amount Union Gas Phase 1 Not applicable $30,000 per builder in services, education, and training Phase 2 Not applicable $30,000 per builder in services, education, and training Phase 3 Not applicable $15,000 per builder in services, education, and training Ongoing Engagement Not applicable $17,500 per builder in services, education, and training First-time IDP Participation Not applicable $2,000/qualifying home built (up to 50 homes) Second-time IDP Participation Not applicable $1,000/qualifying home built (up to 100 homes) Third-time IDP Participation Not applicable $500/qualifying home built (up to 200 homes) Commercial Savings by Design The Commercial Savings by Design program provides builders and designers of new, Part 3 commercial buildings with support during the building design process including an Integrated Design Process (IDP) DNV GL 11/10/16 Page 164

180 Session with support from industry experts, conservation authorities and municipalities as well as financial support to cover the costs associated with the IDP and performance incentives for achieving savings above code. As a market transformation program, there are no specific savings associated with the program. Measures, incentives, and savings: Commercial Savings by Design Measure Description Savings per Measure Incentive Amount Integrated Design Process Not applicable Up to $30,000 value per project Energy Performance Target Not applicable $15,000 incentive when target has been met Final As-Built Energy Model & Commissioning Report Union Not applicable $15,000 per builder when model and report have been submitted and approved Financial Incentives & Design Support Not applicable It is anticipated that Union will have similar offerings as, however the specific amounts were not available. Run it Right () Run Smart (Union) The Union Gas Run Smart program provides incentives to eligible customers for the successful implementation of low-cost or no-cost energy savings measures and activities. Eligible customers include Education, Healthcare, Office, Multi-Unit Residential and Entertainment Commercial customers using at least 50,000 m 3 of natural gas per year who have not participated in a Union Gas energy saving program in the last two years. As market transformation programs, there are no specific savings associated with the programs, however financial incentives are available to participants as follows. DNV GL 11/10/16 Page 165

181 Measures, incentives, and savings: Run It Right and Run Smart Measure Description Savings per Measure Incentive Amount Investigation Phase Not applicable 100% of cost Implementation Phase Not applicable Up to $10,000 to implement operational improvements Monitoring Phase Not applicable 100% of the cost of 12 months of EMIS Union Implementation of low-cost/no-cost energy savings measures Not applicable $0.20/m 3 to $0.30/m 3 to eligible participants Comprehensive Energy Management () Strategic Energy Management (Union) The CEM program includes five components: first, establishing an energy management sponsor at the senior management level; second, the development of an energy policy with technical support from ; third, the identification of an energy management champion; fourth, the development of an energy model with technical support and financial assistance from and fifth, the implementation of a continuous improvement approach to energy improvement. Under Union s program, large commercial and industrial customers receive financial incentives in order to purchase and install energy data management and sub-metering equipment. These same customers also receive training and technical support for proper use of the energy monitoring software. As market transformation programs, there are no specific savings associated with these programs, however support is provided as discussed above and the utilities have been ordered by the OEB to add a savings metric to the program by mid-term. DNV GL 11/10/16 Page 166

182 Measures, incentives, and savings: Comprehensive Energy Management and Strategic Energy Management Measure Description Savings per Measure Incentive Amount NA Not applicable NA Union Year 1 Not applicable Up to $25,000 for sub-metering and data management system Year 2 Not applicable In-kind technical support Years 3-5 Not applicable Year 3: $10,000 Year 4: $15,000 Year 5: $20,000 upon achieved savings from baseline School Energy Competition () The School Energy Competition program offers technical assistance (such as workshops and curriculum), training within schools (including social marketing and webinars), and marketing to increase engagement in conservation activities. The program offers a web-based energy management information system and up to $12,000 in prizes for the first, second, and third placed schools in the annual competition. As a market transformation program, there are no specific savings associated with the offering. Energy Leaders The Energy Leaders program offers technical assistance (such as site assessments and system audits), training (such as webinars and in-person design workshops), and communication (such as case studies and direct mail campaigns) to assist participants who seek to achieve deep savings or install emerging technologies. The program offers incentives that are up to 50% higher than existing offers. As a market transformation program, there are no specific savings associated with the Energy Leaders offering. DNV GL 11/10/16 Page 167

183 APPENDIX F. Process Flow This appendix contains a short description of the process by which each program is delivered, where available. Home Energy Conservation () Home Reno Rebate (Union) The programs are offered to the consumer by and Union through certified energy advisors. Participants call a participating certified energy advisor to receive a pre-retrofit home energy evaluation. Following the evaluation, retrofit work is completed by a qualified contractor and/or by the participant him/herself. Once the retrofit work is completed, a post-retrofit home energy evaluation by a certified energy advisor is conducted and used to calculate gas savings. Residential Adaptive Thermostats () The program is offered directly by and through participating manufacturers. To be eligible for a rebate, customers must meet the following criteria: Residential customer in franchise area Valid EGD account number Proof of purchase and installation of the unit. The rebate is paid as a bill credit on the account. Customers apply for the credit through the manufacturer s website. Home Winterproofing () Home Weatherization, Furnace End-of-Life, and Aboriginal Offerings (Union) Participation follows these steps: The participant fills out an application. The application is reviewed to determine eligibility. A delivery agent visits the home and assess potential improvements. Professional contractors install insulation and draft proofing measures. After the work is complete, a delivery agent conducts a post-retrofit home energy evaluation. Participants must participate in pre- and post-renovation assessments to assess savings, using pre and post energy modelling software (HOT2000 in EnerGuide mode). Reports are submitted from delivery agents summarizing installation site information and natural gas savings calculated based on the results of customized home energy evaluations. The eligibility requirements for both programs are based on Statistics Canada Low Income Cut-Off ( LICO ) plus 35% based on community size greater than 500,000 (i.e., households with income at or below 135% of LICO qualify). The LICO values are updated by Statistics Canada annually, and adjusted for program purposes accordingly. DNV GL 11/10/16 Page 168

184 As an alternative to the household income criteria, customers can participate if they received one of the following social benefits within the last twelve months 36 : The National Child Benefit (NCB) Allowance for the Survivor Guaranteed Income Supplement (GIS) Allowance for Seniors Healthy Smiles Ontario Child Dental Program Ontario Works Ontario Disability Support Programs (ODSP) LEAP Emergency Financial Assistant Grant Participants of electric CDM Home Assistance Program In addition to income eligibility, customers must pay their gas bill. Low Income Multi Residential Affordable Housing () Low Income Multi-Family Offering (Union) outlines the following eligibility criteria for the Affordable Housing offer: Part 3 Buildings owned and operated by social housing providers as well as privately owned buildings identified as low income Privately owned low income buildings will be municipally informed and census defined. Additional eligibility requirements for private sector participants will be the subject of discussions between the low income working group of interveners and stakeholders, and Social housing and assisted housing buildings as described in the Housing Reform Act of 2011 and DSM Framework The eligibility criteria for Union are as follows: Social and Assisted Housing Social and Assisted Housing Providers that meet the following criteria will be eligible: Operate Part 3 buildings with tenants who have a household income at or below 135% of the most recent Statistics Canada Pre-Tax Low-Income Cut-Offs (LICO) for communities of 500,000 or more, as updated from time to time (income eligibility to be confirmed by the housing provider). Low-Income Market Rate Multi-Family Privately owned multi-family, Part 3, buildings that have a high propensity of low-income tenants (Union uses multiple data sources), and average rents of the building are at or below the Average Market Rent for that municipality. Savings for each project are determined with project-specific savings calculation. Where applicable, ESCs utilize standardized engineering calculations. 36 Union also accepts participation to End-of-Life Furnace Upgrade program as proof of eligibility for Home Weatherization. DNV GL 11/10/16 Page 169

185 Large Volume (Union) Union s Large Volume program is aligned under one brand platform, EnerSmart. Large Volume custom projects are jointly delivered through Union s Account Managers and Project Managers. This approach is aimed at influencing the market and achieving successful implementation of the program. The Account Manager s role is to work with assigned customers. The Project Managers work together with the Account Managers, as well as third party engineers, equipment manufacturers, and service providers as necessary to complete custom applications. As part of the QA/QC process for the Large Volume program, each custom project is assessed by Union s internal project review and verification QA/QC team prior to an external verification and audit. The objective of the QA/QC process is to confirm the calculated savings through evaluation of project and customerspecific factors including: Reasonableness of base case assumptions; Confirmation of high efficiency case assumptions; Reasonableness of project life assumptions (EUL); Confirmation of other factors affecting gas demand (e.g. production and weather); and Confirmation of customer project costs. Commercial and Industrial Prescriptive Energy Solutions () Prescriptive and Quasi-Prescriptive Offering (Union) commercial prescriptive offer provides service to over 150,000 Commercial sector customers across their franchise area. These customers are segmented across sub-sectors, which include: multi-residential (not including social housing), office buildings, schools/universities, hotels/motels, warehouses, retail, food services, hospitals/health-care Facilities and government/municipal. The program targets commercial customers who are primarily in Rate 6 as well as those in customers in Rates 135, 145, 110, 115 and 170. For the Commercial Prescriptive programs, engineering studies are used to develop estimated savings associated with the installation of a particular device, e.g., a low flow showerhead. The resulting savings estimates are applied to every unit installed as part of the program offering. The offer is primarily promoted and delivered by Energy Solutions Consultants (ESCs) who are active in the marketplace and work directly with customers, meeting with building operators and facility managers to conduct site visits and make recommendations based on each building s unique systems. ESCs work with national chain and large property management firms to introduce savings strategies and align DSM offers with the customers long term energy plans. ESCs use their technical expertise to work with smaller firms and managers of standalone buildings by educating them on savings concepts and providing recommendations and savings estimations for potential projects. Union prescriptive and quasi prescriptive offering Union delivers their program using targeted marketing to individual segments, which allows them to tailor their communication to the market and barriers for each segment. In addition to the general program, Union offers the following targeted services: DNV GL 11/10/16 Page 170

186 National Account customers are those that have multiple property locations throughout Union s franchise with similar design and use, such as retail chains, property management firms, and food service chains. Union offers these customers a multi-unit installation bonus. Hotel and motel customers are offered an additional incentive for participating in the Ozone Laundry initiative to overcome barriers related to understanding the technology s benefits. To reach C&I customers, Union implements a combination of the following approaches: Direct Sales Approach. With this approach, Union works directly with the end-use customer to provide education on potential options to improve the energy efficiency of their facilities, offerings available to facilitate those options, and how the application process works. Mass Market Approach. Union uses a number of mass marketing techniques to target the end-use customer such as the Union webpage, bill inserts, direct mails, blasts, and advertising. Event-based Marketing. Union also uses event-based marketing including tradeshows, customer workshops, and other similar events to reach a large number of customers and industry partners. National Account Approach. Union s National Account managers work with customers who make decisions using a top-down, centralized approach. Union also provides support through a network of industry partners who specify or install energy efficient equipment and/or directly educate or influence Union s customers to adopt natural gas energy efficient equipment. Commercial and Industrial Custom Energy Solutions () Custom Offering (Union) C&I custom offer The C&I Custom program targets commercial customers who are primarily in Rate 6 as well as those in customers in Rates 110, 115, 135, 145 and 170, and provides incentives for customers undertaking capital and operational improvements. Typical measures include boiler and HVAC retrofits, controls and building automation systems, heat recovery projects and building envelope improvements. The C&I Custom offer is primarily promoted and delivered by Energy Solutions Consultants (ESCs) who are active in the marketplace. C&I Custom ESCs work directly with customers, conducting site visits and making custom recommendations based on each building s unique systems. They also work with national chain and large property management firms to introduce savings strategies and align DSM offers with the customers long term energy plans. ESCs educate smaller firms and managers of standalone buildings on savings concepts and provide recommendations and for potential projects. Savings for each C&I Custom project are calculated on an individual basis and then tracked monthly by the Tracking and Reporting team, utilizing EGD s sales tracking software. Union custom offering The Union Custom offering includes the following components: Customer communication and education Engineering feasibility and process improvement studies Operation and maintenance improvement incentives New equipment and processes incentives DNV GL 11/10/16 Page 171

187 Energy management incentives, including energy meters and monitoring and management systems Union uses standard calculators to estimate natural gas savings for a selection of commercial custom projects. The standard calculators used in 2015 were: Formula 1 laundry Hot water heating Destratification fan Roof insulation Make up air VFD retrofit Boiler combustion control Make up air High extraction washer (>300G) Window Dock door seals Each Union custom project is assessed by Union s internal project review and verification Quality Assurance/Quality Control ( QA/QC ) team prior to external verification and audit. The review is conducted by engineers within Union s custom C&I project team. The custom project team reviews and confirms the calculated savings through evaluation of project and customer specific factors including: Reasonableness of base case assumptions; Confirmation of high efficiency case assumptions; Reasonableness of project life assumptions (EUL); Confirmation of other factors affecting gas demand (e.g. production and weather); and Confirmation of customer project costs. Run it Right () Run Smart (Union) s Run It Right program is available to customers with access to daily consumption data via an existing Metretek meter. The program is implemented via four stages as shown in the following figure. DNV GL 11/10/16 Page 172

188 Run It Right Program Process Source: EB Multi-Year Demand Side Management Plan ( ) School Energy Competition () Process flow for School Energy Competition Program Source: EB Exhibit B, Tab 2, Schedule 1, page 84 DNV GL 11/10/16 Page 173