United Nations Development Programme. Ministry of Energy and Natural Resources General Directorate of Renewable Energy Government of Turkey

Transcription

1 United Nations Development Programme Ministry of Energy and Natural Resources General Directorate of Renewable Energy Government of Turkey Evaluation of UNDP/GEF Project: Market Transformation of Energy Efficient Appliances in Turkey (EVÜdp) (PIMS No: 4014) Mid-Term Evaluation Report Mission Members: Mr. Roland Wong, International Consultant Mr. Koray Abaci, National M&E Administrator July 2012

2 TABLE OF CONTENTS PAGE ACKNOWLEDGEMENTS... II ABBREVIATIONS... III EXECUTIVE SUMMARY...V 1. INTRODUCTION BACKGROUND RATIONALE FOR PROJECT PROJECT GOALS, OBJECTIVES AND EXPECTED RESULTS MID-TERM EVALUATION Purpose of the Evaluation Key Issues to be Addressed Evaluation Methodology and Structure of the Evaluation PROJECT IMPLEMENTATION ARRANGEMENTS KEY FINDINGS PROJECT PROGRESS AND ACHIEVEMENTS TO DATE Project Outputs Project Impacts PROJECT DESIGN AND RELEVANCE Project Relevance and Country Drivenness Project Design and Implementation Approach PROJECT IMPLEMENTATION ARRANGEMENTS Stakeholder Involvement, Linkages to Project and Other Interventions in Sector Management, Monitoring and Evaluation, Identification and Management of Risk PROJECT BUDGET AND COST EFFECTIVENESS EVALUATION OF PROJECT SUSTAINABILITY AND REPLICABILITY Sustainability Replicability CONCLUSIONS AND RECOMMENDATIONS CONCLUSIONS RECOMMENDATIONS LESSONS LEARNED APPENDIX A MISSION TERMS OF REFERENCE APPENDIX B MISSION ITINERARY (FOR MAY 22 TO JUNE 1, 2012) APPENDIX C LIST OF PERSONS INTERVIEWED AND DOCUMENTS REVIEWED APPENDIX D S&L MARKETING MONITORING CALCULATIONS FOR GHG REDUCTIONS APPENDIX E REVISED PROJECT LOGICAL FRAMEWORK JANUARY 2011 INCEPTION REPORT Mid-Term Evaluation Mission i July 2012

3 ACKNOWLEDGEMENTS The Evaluators wishes to acknowledge with gratitude the time and effort expended by all project participants and stakeholders during the evaluation interviews. This provided valuable insights, candid perspectives, and made the evaluation process more enjoyable for the entire team. In particular, we wish to thank UNDP Turkey for arranging mission logistics, itinerary and stakeholder interviews. We hope that this report will contribute to the successful conclusion of the project, and the sustained trend of increased use of energy efficient appliances, and reduced electricity consumption in Turkey. Mid-Term Evaluation Mission ii July 2012

4 ABBREVIATIONS AGID APR CDM CO DEX DGRE DNA EBRD EE EIE EOP ESCO EU EVÜdp GDRE GEF GHG GoT IPA ISKID KfW ktoe kwh log-frame M&E MoENR MoEU MoD MoF MFA MoSIT MTE Mtonnes MW MWh NGO PIR ProDoc PIR PMSP PMU PSC RE RTA S&L Turkish Lighting Luminaires Manufacturers Association Annual Progress Report Clean Development Mechanism Country Office Direct Execution (of a project by UNDP) Directorate General of Renewable Energy (under MoENR) Designated National Authority European Bank for Reconstruction and Development Energy efficiency General Directorate of Electrical Power Resources Survey and Development Administration End-of-Project Energy Service Company European Union Market Transformation of Energy Efficiency Appliances Project in Turkey General Directorate of Renewable Energy (under MoENR) Global Environment Facility Greenhouse gas Government of Turkey Instrument for Pre-Accession (under EU Commission assistance) Air Conditioning and Refrigeration Manufacturers Association in Turkey Kreditanstalt für Wiederaufbau / German Bank for Reconstruction Kilotonnes of oil equivalent Kilowatt hour logical framework matrix Monitoring and Evaluation Ministry of Energy and Natural Resources Ministry of Environment and Urbanization Ministry of Development Ministry of Finance Ministry of Foreign Affairs Ministry of Science, Industry and Trade Mid-Term Evaluation million tonnes Megawatt Megawatt hour Non-governmental Organization Project Implementation Report UNDP Project Document for Market Transformation of Energy Efficient Appliances Project in Turkey Project Implementation Reports Proactive Market Surveillance Program Project Management Unit Project Steering Committee Renewable Energy Regional Technical Advisor Standards and Labeling Mid-Term Evaluation Mission iii July 2012

5 SME TA TSE TESID TURKAK TURKBESD UNDP USAID UNFCCC Small and Medium Size Enterprise Technical Assistance Turkish Standards Institute Turkish Electronics & Information Industries Association Turkish Accreditation Agency Turkish White Good Manufacturers Association United Nations Development Programme U.S. Agency for International Development United Nations Framework Convention on Climate Change Mid-Term Evaluation Mission iv July 2012

6 EXECUTIVE SUMMARY Brief Description of the Project The UNDP-GEF Project Market Transformation of Energy Efficient Appliances in Turkey (MTEEA or Project) aims to reduce GHG emissions through increased sales of energy efficient appliances that include refrigeration equipment, dish washers, washing machines and driers, electric ovens, air conditioners and televisions. To achieve this goal, MTEEA was designed to improve the institutional capacities for the development and implementation of effective EE appliance policies, to strengthen existing enforcement and verification programmes designed to improve compliance with EE policies, and to strengthen awareness of EE appliances through the entire supply chain (i.e. local manufacturers, wholesalers, retail outlets, consumer groups), and disseminating lessons learned. The EE Appliances Project has been operational for 18 months, with the Inception Mission taking place in December MTEEA is implemented under a NEX modality, under the management of the Executing Agency, the General Directorate of Renewable Energy or DGRE (formerly General Directorate of Electrical Power Survey and Development Administration, EIE). DGRE s relationship with other agencies on this Project includes: The General Directorate of Industry in MoSIT 1, which is in charge of transposing EU S&L regulations into Turkish legislation and the General Directorate for Safety and Inspection of Industrial Products, which is responsible for market surveillance under EU S&L regulations. A significant portion of the capacity building activities of MTEEA were proposed under MoSIT; The Turkish Standards Institute (TSE), which is transposing EU test procedures into a Turkish context; The Turkish Accreditation Agency (TURKAK), a financially and administratively autonomous agency responsible for accrediting local laboratories, inspection and certification services to EU standards. In addition, non-governmental groups are critical to MTEEA including: The Association of White Goods (TURKBESD), which provides sales data for the EE appliances product groups within the scope of the Project ; Arçelik, a private company that has numerous retail outlets and is a willing partner in raising awareness of the S&L drive for EE appliances. With the project starting operations on December 6, 2010, the MTE has evaluated approximately 18 months of Project progress, and is providing recommendations for the remaining period of the Project scheduled to end in December Context and Purpose of the Evaluation The purpose of the mid-term evaluation (MTE) for this Project was to evaluate the progress towards attainment of global environmental objectives, project objectives and outcomes, capture 1 The Ministry of Industry and Trade (MoIT) was reorganized as the Ministry of Science, Industry and Technology (MoSIT) as a result of a Decree Law published in the Turkish Official Gazette on 4 June Mid-Term Evaluation Mission v July 2012

7 lessons learned and suggest recommendations on major improvements. The MTE is to serve as an agent of change, play a critical role in supporting accountability, and serve to: Strengthen the adaptive management and monitoring functions of the Project; Enhance the likelihood of achievement of Project and GEF objectives through analyzing project strengths and weaknesses and suggesting measures for improvement; Enhance organizational and development learning; Enable informed decision-making; Create the basis for replication of successful project outcomes achieved to date; Identify and validate proposed changes to the project document to ensure achievement of all project objectives; and Assess whether it is possible to achieve the objectives in the given timeframe, taking into consideration the speed at which the project is proceeding. Evaluation of Project The overall rating of the Project is Satisfactory (S), based mainly on: Satisfactory progress on all outcomes at the mid-point of the Project, with few or no delays relative to the original schedule. This includes timely and effective delivery of the transposition of EU legislation, and knowledge transfers to MoSIT officers on compliance checking and enforcement; Current preparations for numerous training workshops in 2012 that will strengthen an excellent foundation for market transformation for the remaining period of the Project. Current preparations include sales and inspector training on EE issues, pilot testing of the PMSP, and implementation of an awareness programme on EE appliances; Less than 16% of resources spent to date at the mid-term of the Project, translating into resources being available for valuable capacity building activities for the remainder of the Project attached to a high degree of confidence remaining resources will be and are in the progress of being expended; Table A: Summary Evaluation of Project Project Strategy Relevance Efficiency Outcome 1: Enhanced institutional capacities to develop and implement effective appliance EE policies Outcome 2: A structured enforcement and verification programme with adequately trained staff Outcome 3: Raised awareness of the end-users and supply chain and strengthened capacity of the local manufacturers Outcome 4: Institutionalization of the support provided by the project, including monitoring, learning, adaptive Effectiveness Overall Rating S MS S S S S S S S S Unable to rate Unable to rate Unable to rate Unable to rate Unable to rate Unable to rate feedback and evaluation Monitoring and Evaluation S S S S Overall Rating S S S S Mid-Term Evaluation Mission vi July 2012

8 Project sustainability rating is Likely due to: The strong engagement of all stakeholders interviewed with the Project; A strong foundation built for market transformation that includes: EE being a high government priority; the transposition of EU Energy Labeling and EcoDesign regulations into Turkish; the Project plan for market surveillance system being accepted by government; strong commitments of government to adopt a Project-designed market monitoring system database; strong commitments for field testing of equipment; and excellent attendance at training sessions for field inspectors. Conclusions The Project has laid an excellent foundation for market transformation based on the progress of the EU legislation transposition, the design and implementation of a market surveillance system, the training of field inspectors to check compliance and enforce standards, and the progress of work achieved thus far in setting up of testing laboratories to implement the compliance checking programmes. One issue that needs attention during the remainder of the Project is the quality of energy consumption data received from industry associations. Industry associations are currently only reporting appliances sold by energy classes. Currently, TURKBESD and other associations are reporting energy consumptive data for appliances where sales data was available for appliances complying with recently developed energy labels such as refrigerators, freezers and washing machines. Reductions have not been reported for other appliances due to energy labels being recently adopted (dishwashers), or under development (stove ovens). In the case of air conditioners, no energy label has yet been adopted due to the need to further disaggregate the appliance type into cooling and heating/cooling types. For televisions, TURKBESD has not collected sufficient sales data of labeled televisions. To improve the accuracy for reporting more accurate energy consumption figures for appliances, sales figures should be reported as a weighted average of each appliance (each appliance has a specific energy consumption figure). A barrier to this is due to manufacturer s fears of revealing information that would affect their competitiveness with other name brands. With the progress achieved thus far, there is a strong likelihood that the Project completion date will extend beyond December 31, 2013 due to: Lengthy time required to complete activities for Outcome 3: Design of promotional campaigns and financial incentives to accelerate EE appliance market transformation. With the time required to formulate and design a promotional campaign to phase-out old, inefficient appliances, one significant issue is how to manage the appliance waste given that there is no integrated waste management strategy for old, inefficient appliances. A recycling programme for waste metal from old appliances will be a necessity for such a programme to work. More than 18 months will be required to obtain acceptance of such re-cycling measures with the manufacturer and relevant government agencies; Mid-Term Evaluation Mission vii July 2012

9 Piloting the PMSP and the arrangements to test equipment (as related to Output 2.4). The TSE will require more than 18 months to upgrade its testing laboratories for air conditioners, as it will: o require clarification on specifications of air conditioners to be tested that may require another 6 to 12 months to decide on the required air conditioner tests; o need to design and estimate costs for an upgraded laboratory with new equipment over a 6 to 8 month period; and o require approximately 12 months to physically upgrade the laboratory. Further changes to government institutions. This is a risk beyond the control of the Project that can be partially mitigated by the Project Director, who will be monitoring planned changes to the relevant institutions. Recommendations Recommendation 1: Improve quality of energy and GHG data received from industry associations and other sources: MoSIT, with guidance from MoENR and MoEU, will need to provide specific direction to industry associations and manufacturers on reporting sales on the basis of weighted average energy consumption for each product group. This will involve using sensitive product sales data which does not need to be disclosed during the reporting process. Such an effort would be in parallel with MoEU s new regulation of April 2012 obliging certain facilities to establish annual monitoring, reporting and verification (MRV) of greenhouse gas (GHG) emissions by January The Government has designated TURKAK 2 (Turkey s sole accreditation agency) as its third party to approve a facility s MRV plans; Private market research companies may need to be employed to improve data collection and quality of data as a backup plan in the event there are reporting compliance problems with the various manufacturers and industry association representatives; With MoEU being responsible for reporting GHG emissions to the UNFCCC, it also needs to be involved with quality assurance of GHG data reported to it in close collaboration with TURKAK. TURKAK is now designing the installation-level monitoring systems to EU-ETS standards; Recommendation 2: Explore possible financial incentives to accelerate EE appliance market transformation such as: A government subsidy for the purchase of an EE appliance. Such a request needs to be backed by a good business case for the government, and may require the assistance of the EE Appliances Project, as directed by DGRE and the appliance manufacturers; Linking appliance purchases with the country s voluntary carbon finance markets, with the replacement programme designed with assurances that the old appliances are exchanged for new appliances; and An ESCO business model. This model may not work for household appliances, but could be applied to large industrial-scale installations, such as central air conditioners and heaters. This is likely outside the scope of the Project. 2 Mid-Term Evaluation Mission viii July 2012

10 Recommendation 3: The MTEEA Project needs to leave open the possibility of extending the project terminal date to sometime in 2014 to allow the project sufficient time to achieve its objectives. The MTE mission feels that the Project will likely take more than 18 months to complete the following outputs: Set-up of financial incentives (related to Outcome 3). A likely scenario is to have each manufacturer with its own programme linked with a voluntary carbon scheme. This may take more than 18 months to establish, depending on how quickly the manufacturers adopt their sales to the carbon scheme; Build-up of TSE testing laboratories (notably with air conditioners) and the subsequent agreements and arrangements of MoSIT to use TSE on a long-term basis for testing of the 6 groups of appliances covered under the project. TSE will likely need more than 18 months to get air conditioner testing equipment, including set-up of the plan of which air conditioners to test (split, wall units and for the type of climatic zone), procurement of testing equipment, installation and commissioning of equipment. Lessons Learned Two valuable lessons from the MTEEA Project are as follows: GEF is a window for recipient countries to international expertise that can provide the best international practices. If used properly, GEF international consultancy contracts can provide an excellent foundation for project implementation; Adaptive management is important to improve the likelihood of a project achieving its intended outcomes. The Project, thus far, has demonstrated adaptive management after the transposition of the new EU Directives on Eco-Design and Energy Labeling. At this time, training needs were identified for MoSIT field inspectors in the area of energy efficiency and how to ensure compliance and enforce transposed EU Directives. In response to the high interest shown by government officials, MTEEA project management organized and provided the requested training with the MTE Mission observing that the trainees have embraced the training process. MoSIT demand for the training has increased to the extent that initial sessions are designed to train-thetrainers and to cover topics ahead of schedule such as Management of Market Surveillance Programme. Mid-Term Evaluation Mission ix July 2012

11 1. INTRODUCTION This report summarizes the findings of the Mid-Term Evaluation Mission conducted during May 22-June 1, 2012 for the UNDP-GEF project entitled Market Transformation of Energy Efficient Appliances Project for Turkey (herein referred to as the Project or MTEEA, PIMS 4014) implemented by the United Nations Development Programme (UNDP) with financing support provided by the Global Environment Facility (GEF). The Project Document (ProDoc) provides details on reducing household electricity consumption and GHG emissions in Turkey through accelerating market transformation of less energy consuming appliances covered by the Project include refrigeration equipment, dish washers, washing machines and driers, electric ovens, air conditioners and televisions. Project activities include: a) strengthening of local institutions to develop, adopt and implement policies for EE appliances; b) developing a structured compliance checking and enforcement programme for appliance energy standards and labeling; c) increasing awareness and capacity of the consumer and supply chain to purchase and deliver energy efficient appliances in the Turkish market; and d) reporting and sharing project results for learning, adaptive management and possible replication in other countries. The Project commenced operations in March 2010 with the conclusion of the Inception Phase that was marked by an Inception workshop held in December The Project completion is scheduled for December 31, Background The improvements in the standards of living in Turkey over 10 years have increased the country s energy consumption and GHG emissions. Turkey s GHG emissions have increased from 225 Mtonnes CO 2eq in 2000 to over 299 Mtonnes CO 2eq in 2009, an increase of 33% 3. Emissions from electricity generation have increased from 26 Mtonnes CO 2eq in 1990 to over 83 Mtonnes CO 2eq in 2009, a three-fold increase. Electricity generation comprises over 28% of Turkey s GHG emissions. Since 1987, nearly all Turkish households have had access to electricity. With rising standards of living, the average Turkish household has had increasing access to electrical appliances such as refrigerators, ovens, washing machines, driers and air conditioners. As a result, average household electricity consumption in Turkey has risen from 1,518 kwh in 2000 to 2,010 kwh in 2008, an average rise of 3.6% over an 8-year period 4. Prior to the commencement of the Project, the government had undertaken measures to steer appliance sales towards more modern, less energy-consuming household appliances since the 1990s; however, the government s ability to regulate, monitor and enforce the sale and use of these appliances was limited. This has led to products on the market that do not meet EU standards for energy performance 5, a loss of consumer confidence in the purchase of these appliances, and, consequently, a lower market share GDRE estimates that there are over 22 million energy inefficient appliances currently in use in Turkey. Mid-Term Evaluation Mission 1 July 2012

12 1.2 Rationale for Project Turkey s recognition of the Kyoto Protocol on 7 February 2009 signified the government s plans to implement a multi-pronged strategy for reducing its energy consumption. In its efforts related to accession to the EU, the government has recognized the need to harmonize and transpose EU regulations and policies on energy performance of household appliances on the market. A number of barriers and constraints, however, were identified as being impediments to Turkey s harmonization with EU policies and regulations, including: Limited availability of market information on the sale of new appliances with respect to their energy classes and state of origin; Lack of an effective compliance and enforcement system that can test random samples of market products for compliance with minimum energy performance standards and has the legal force to ensure compliance with these standards; Insufficient procedures to remove non-compliant products that would encourage manufacturers to supply products in full compliance with adopted labels for minimum energy performance standards; and Limited knowledge and experience of public sector officials on market monitoring and compliance checking schemes that have worked in other countries. Against this backdrop, MTEEA was designed in 2009 to remove these barriers and constraints to market transformation of energy efficient appliances in Turkey through a standards and labeling (S&L) approach that is intended to reduce household electricity consumption and associated GHG emissions. Project activities proposed to facilitate this objective include: Strengthening the local institutional capacity to develop, adopt and implement effective appliance EE policies; Developing and implementing a structured compliance checking and enforcement programme for appliance energy performance labels and standards; Increasing consumer and the supply chain awareness and capacity to purchase / deliver energy efficient appliances in the Turkish market; and Analyzing and reporting the results of the project for further learning, adaptive management and, as applicable, replication in other countries. Through its association with its co-financing partners, the Project was also designed to leverage additional financing for specific promotional programmes to: Encourage consumers to select energy efficient appliances that go beyond the adopted minimum energy performance standards (if any); Use the appliances in an energy-efficient way during their operation; and Expedite the replacement of old inefficient appliances that have more than years of service. 1.3 Project Goals, Objectives and Expected Results The project development goal is to reduce household electricity consumption and related greenhouse gas emissions of Turkey by accelerating and ensuring the market transformation towards more energy efficient appliances. Mid-Term Evaluation Mission 2 July 2012

13 To achieve this goal, the Project was designed to achieve a number of outcomes 6 : Enhanced institutional capacities in Turkey to develop and implement effective appliance EE policies. This was to be achieved through the activities to enhance the capacity of public officials to implement and monitor S&L laws and regulations, developing market monitoring systems, and forming alliances with the private sector to implement voluntary schemes and agreements that would have the impact of accelerating replacement of old appliances; A structured enforcement and verification programme with adequately trained staff and other resources. This was to be achieved through the development of agreements to consolidate compliance checking and enforcement schemes for products and retailers on EE aspects, upgraded procedures and institutional arrangements for EE testing of products, field testing of checking and compliance schemes, and training staff of testing laboratories and MoSIT state inspectors; Raised awareness of the end-users and the supply chain and strengthened capacity of local manufacturers to develop and implement specific promotional activities to enhance the sale of energy efficient appliances. This was to be achieved through conducting of baseline and post-programme surveys to assess awareness levels and key consumer drivers for the purchase of EE appliances, conducting joint marketing campaigns with manufacturers and retailers on the benefits of EE appliances and lifecycle cost approaches, a dedicated website on consumer choices of appliances and test results of various EE appliances, training of sales staff in retail chains, and implementing specific campaigns for the phase-out of old appliances Institutionalization of the support provided by the project, including monitoring, learning, adaptive feedback and evaluation. This was to be achieved through updating baseline studies to measure the true impact of the Project, infusing EE into educational curricula, elaboration of financial support mechanisms to accelerate the shift towards EE appliances, and reporting on the lessons learned from the implementation of the Project. 1.4 Mid-Term Evaluation Purpose of the Evaluation The purpose of the mid-term evaluation (MTE) for this Project was to evaluate the progress towards attainment of global environmental objectives, project objectives and outcomes, capture lessons learned and suggest recommendations on major improvements. The MTE serves as an agent of change and plays a critical role in supporting accountability. As such, the MTE serves to: Strengthen the adaptive management and monitoring functions of the Project; Enhance the likelihood of achievement of Project and GEF objectives through analyzing project strengths and weaknesses and suggesting measures for improvement; Enhance organizational and development learning; Enable informed decision-making; 6 Based on the Project s Inception Report of May 2010 Mid-Term Evaluation Mission 3 July 2012

14 Create the basis for replication of successful project outcomes achieved to date; Identify and validate proposed changes to the ProDoc to ensure achievement of all project objectives; and Assess whether it is possible to achieve the objectives in the given timeframe, taking into consideration the speed, at which the project is proceeding. In accordance with UNDP/GEF monitoring and evaluation (M&E) policies and procedures, all projects with long implementation periods (e.g. over 3 or 4 years) are strongly encouraged to conduct MTEs. In addition to providing an independent in-depth review of implementation progress, the MTE is intended to be responsive to GEF Council decisions on transparency and better access to information during implementation. MTEs are intended to identify potential project design problems, assess progress towards the achievement of objectives, identify and document lessons learned (including lessons that might improve design and implementation of other UNDP/GEF projects), and to make recommendations regarding specific actions that might be taken to improve the project. It is expected to serve as a means of validating or filling the gaps in the initial assessment of relevance, effectiveness and efficiency obtained from monitoring. The MTE provides the opportunity to assess early signs of project success or failure and prompt necessary adjustments. For these reasons, an evaluation mission was fielded to Ankara and Istanbul from May 22- June 1, 2012 for the MTE of this UNDP-GEF full-sized Project Key Issues to be Addressed Key issues to be addressed by this MTE include: The appropriateness of the project concept and design in the context of current events in Turkey; Implementation of the Project in the context of effectiveness and efficiency in the delivery of its activities; and Project impacts based on current outputs and outcomes and the likelihood of sustaining project results. Outputs from this MTE have been used to chart future directions for this Project Evaluation Methodology and Structure of the Evaluation The methodology adopted for this evaluation includes: Review of project documentation (i.e. project documents, APRs/PIRs, inception meeting minutes) and other pertinent background information; Interviews with key project personnel including the Project Manager, past project personnel, project consultants, and relevant UNDP staff; Interview with relevant stakeholders from Government (e.g. DGRE, MoSIT, MoNRE, GEF focal point); and Field visits to selected project sites and interviews with beneficiaries. Mid-Term Evaluation Mission 4 July 2012

15 A detailed itinerary of the Mission is shown in Appendix B. A full list of documents reviewed and people interviewed is given in Annex C. The Evaluation Mission for the UNDP-GEF project comprised one International Consultant and one National Consultant. This evaluation report is presented as follows: An overview of project implementation from the commencement of operations in March 2010; Review of project results based on project design and execution; Conclusions and recommendations that can increase the probabilities of a successful conclusion; and Lessons learned from implementation of the project to date. This evaluation has taken into consideration the GEF Monitoring and Evaluation policy available from: as well as the UNDP-GEF Monitoring and Evaluation policy that can be downloaded from: The Evaluation also meets conditions set by the UNDP Document entitled Handbook on Planning, Monitoring and Evaluating for Development Results, 2009: and the Addendum June 2011 Evaluation : pdf 1.5 Project Implementation Arrangements The Project executing agency was formerly the General Directorate of Electrical Power Resources Survey and Development Administration (EIE). Following national elections held in June 2011 and a subsequent reorganization of government ministries, the Project executing agency is now the General Directorate of Renewable Energy (GDRE) under the Ministry of Energy and Natural Resources (MoENR), which follows UNDP guidelines for nationally executed projects (with UNDP being the GEF Implementing Agency). The day-to-day management of the project has been carried out by a Project Management Unit (PMU) under the overall guidance of the Project Steering Committee (PSC). The PMU is established within the premises of the GDRE and reports to the executing agency and the PSC. The PSC was established to monitor the project progress, to guide its implementation and to support the project in achieving targeted outputs and outcomes. The PSC is composed of Mid-Term Evaluation Mission 5 July 2012

16 GDRE, MoENR, the Ministry of Science, Industry and Trade (MoSIT), UNDP Turkey, the Association of White Goods Manufacturers (BESD) and Arçelik A.Ş. Figure 1: Project Management Organigram UNDP/GEF RCU UNDP CO Project Steering Committee Project Director PMU Consultants and subcontractors Final Beneficiaries Mid-Term Evaluation Mission 6 July 2012

17 2. KEY FINDINGS 2.1 Project Progress and Achievements to Date Project Outputs The Project has been operational for 18 months, with the Inception Mission taking place in December Notwithstanding a slow start to the Project since March 2010, the Project has delivered a number of critical outputs that increase the likelihood of the Project achieving its intended outcomes by the end of the Project, scheduled for December These outputs include the completion of a number of international consulting outputs by May 2012 that have provided a solid foundation on which future Project activities towards market transformation for EE appliances in Turkey can be effected. For Outcome 1: A review of the current Turkish institutional arrangements, regulatory framework and procedures in Turkey for implementing energy S&L and Eco-Design framework Directives (November 2011); An overview of how appliance EE policies were introduced into other EU member states (February 2012); A report of the study tour to the UK (November 2011); New EU Eco-Design and energy labeling regulations have been transposed for all appliances targeted by the Project, namely refrigeration equipment, dish washers, washing machines and driers, electric ovens, air conditioners and televisions; Two Eco-Design regulations (dishwashers and washing machines) have been published by MoSIT. Publication of regulations for air conditioners is still pending; An implementation plan for completing the development of a proactive market surveillance plan and a market monitoring plan (April 2012); Proposal on the structure and arrangements for data collection and management to establish a market monitoring system in Turkey (September 2011); Report on baseline conditions for establishing a market monitoring system for selected electric household appliances with a database that can be regularly updated with information on the annual sale and energy consumption of different appliances as per their energy consumption classes (April 2012); A market monitoring database and calculation tool for energy-consuming household appliances (May 2012); An S&L monitoring calculation sheet for energy consumption of appliances and associated GHG reductions (May 2012); and A users manual for the market monitoring database and calculation sheet (May 2012). For Outcome 2: An overview of the arrangements for conformity assessment and energy performance testing in the EU Member States for the types of appliances targeted by the Project (November 2011); An inventory and capacity assessment of the existing equipment laboratories in Turkey, with recommendations on how these laboratories can be upgraded to conform with EU Energy Labeling and Eco-design Regulations (February 2012); and Mid-Term Evaluation Mission 7 July 2012

18 A report on a proposed Proactive Marketing Surveillance Plan (PSMP) that proposes a volume and sequencing of appliance testing that is based on the recommendations from the inventory and capacity assessment of the Turkish testing laboratories. For Outcome 3: Public awareness raising and marketing strategy document First Main Field Survey for measuring the level of awareness of consumers regarding the energy efficiency of appliances Actual Project outputs are summarized against the Project log-frame in Table Project Impacts Prior to the start of the Project, the government had considered energy efficiency in appliances and other energy-consuming goods to be a top priority as a means of reducing energy demand and to reduce the need for additional electricity generation plants. The basic problem as articulated by the stakeholders has been the lack of focus to facilitate market transformation of energy efficient appliances. Similar to all governments who are managing a myriad of issues, the government has a challenge in focusing on certain issues such as the household appliance market and to have dedicated personnel to achieve the desired market transformation. While there were systems in place for S&L in Turkey for these appliances as early as 2000, a review of the systems during the design phase of the Project in 2008 revealed that S&L systems were not properly functioning in Turkey. This resulted in a lack of consumer confidence and sales of these appliances. Moreover, market conditions for appliances were changing and the systems were not updated to meet current standards for appliance use in Turkey; for the Turkish export market, this has been different as EU import requirements (that are more stringent than Turkish standards) govern. The impact of the Project, thus far, has been to: Provide the necessary focus for the government to accelerate appliance market transformation in Turkey. Without the EE Appliances Project, the government would continue on its business-as-usual (BAU) path, managing a myriad of issues, and not solely focusing on activities that would transform the market. As a result, market transformation of EE appliances would have been implemented at a slower pace; Catalyze useful dialogue between government and private sector manufacturers on the need to transpose the latest EU standards for adoption by government. The impact of this dialogue has been advanced voluntary implementation EE standards by industry, and implementation of EU regulations by industry before publication in the Turkish Official Gazette. As such, the impact of the Project has been an increased pace of transposition of EU standards in 2011 and 2012, notwithstanding the fact that the government had been active with transposition of EU legislation to Turkey since This includes the transposition of EU regulations for 6 appliances (refrigerators, dishwashers, washing machines/driers, electric ovens, air conditioners and televisions) that fall under Eco-Design and energy labeling regulations as well as inspection of these appliances within the framework of the market monitoring system; Mid-Term Evaluation Mission 8 July 2012

19 Provide a solid foundation for EE appliance market transformation through: o its assistance to harmonize Energy Labeling and EcoDesign regulations between the EU and Turkey; o development of a proactive market surveillance plan (PMSP) in 2012 to strengthen monitoring functions for the collection and analyses of data for sales of the 6 appliances that will provide information on the rate and trends of compliance, verification processes for determining whether a product actually performs according to its claimed energy performance value, and the system of enforcement in responding to non-compliance offences; by engaging private sector manufacturers and various industrial associations (such as TURKBESD and ISKID) to report to DGRE on EE appliances sales and MoSIT on their ability to manufacture appliances to changing standards; o development of market monitoring database through which to provide voluntary reporting in compliance with Article 21-2 of the Regulations on Increasing Efficiency in the Use of Energy Resources and Energy (adopted by the government in October 2008) stipulating: The manufacturers and importers of electrical home appliances and boilers used for heating buildings shall notify to the General Directorate, in January every year, the amount of products they sold in the country, on the basis of energy label categories ; o facilitating plans to have MoSIT use TSE to test equipment as part of the proposed PMSP. This would have the impact of increasing demand for TSE services and justification for TSE to strengthen its product testing capacity, specifically for air conditioners, washing machines and dishwashers; o training MoSIT field inspectors, who are now enhancing their knowledge of current EU Eco-Design and Energy Labeling Directives and an overview of market surveillance activities for which they will have regulatory oversight. Mid-Term Evaluation Mission 9 July 2012

20 Table 1: Project Progress Observed in May 2012 (against January 2011 log-frame from Inception Report) Project Strategy (taken from Prodoc) Project Objective: Reduction of household electricity consumption and related greenhouse gas emissions of Turkey by accelerating and ensuring the market transformation towards more energy efficient appliances Measurable Indicators from Prodoc Log-Frame The estimated stock and annual sale of different energy classes of the appliances selected for monitoring Household electricity consumption trend End of Project (EOP) Target Depending on the product category, 2-28% reduction of the average unit energy consumption by 2013 compared to the estimated baseline development Stabilizing or reducing the total electricity consumption of the targeted appliances Status of Delivery as of May 2012 Rating 7 New versions of EU energy labeling and EcoDesign regulations have been transposed with recently published energy labeling of tumble driers to be transposed in All EcoDesign regulations are in force and new energy regulations are ready for publication in the official Turkish gazette. Turkish manufacturers have already started implementing new energy labels introducing new energy classes. Consumer surveys have been completed in 2012 on the use of EE appliances, establishing a baseline energy consumption of household appliances currently in use. The survey also contains data from consumer s energy consumption and costs related on appliance. Baseline energy consumption has been augmented in 2011 by information from private sector associations (TURKBESD, ISKID) Establishing trends in household electricity consumption will require another year or two of consumer surveys 7 Green cell shows completion and successful achievement; Yellow cell shows expected completion by the end of the project; Red cell show poor achievement and unlikely to be complete by end of Project Mid-Term Evaluation Mission 10 July 2012

21 Table 1: Project Progress Observed in May 2012 (against January 2011 log-frame from Inception Report) Project Strategy (taken from Prodoc) Measurable Indicators from Prodoc Log-Frame Amount of reduced CO 2 emissions compared to the projected baseline End of Project (EOP) Target Estimated min. 1.7 Mtonnes of incremental reduction of CO 2 (with a causality factor of 60%) by the appliances sold during the project Status of Delivery as of May 2012 Rating 7 A marketing monitoring system has been completed containing baseline information on appliance energy consumption that can provide the basis for calculating baseline CO 2 emissions. Based on the current data from TURKBESD and other sources, the Project is expected to meet and possibly exceed its 1.7 Mtonnes CO 2 emission reduction target that translates into a causality factor higher than 60% Outcome 1: Enhanced institutional capacities in Turkey to develop and implement effective appliance EE policies The content and status of new policies and programs supporting their implementation New legal and regulatory provisions and supporting compliance checking, enforcement and outreach programs adopted that reflect international best practices Gap analysis prepared by the Project has been agreed by DGRE. The analysis produced proposals for amendments to EE law #5627 and regulation on increasing efficiency in use of energy and energy resources New EU Eco-Design and energy labeling regulations have been transposed for all 6 appliance groups targeted by the Project. Two Eco-Design regulations (dishwashers and washing machines) have been published by MoSIT. Publication of regulations for air conditioners is still pending. Regulations on energy labeling for all targeted appliances (except for tumble driers which EU published 9 May 2012) have been finalized and delivered to MoSIT, ready for publication. This initiative is being voluntarily implemented by private industry Mid-Term Evaluation Mission 11 July 2012

22 Table 1: Project Progress Observed in May 2012 (against January 2011 log-frame from Inception Report) Project Strategy (taken from Prodoc) Output 1.1: Enhanced capacity of public authorities to implement and monitor the impact of the adopted S&L related laws and regulations, and assess the impact, applicability and required implementation support of possible new regulations and policies. Measurable Indicators from Prodoc Log-Frame The status and type of capacity building provided End of Project (EOP) Target Trained staff and supporting studies to assess the applicability and required implementation support of new regulations and policies, and to monitor and assess the impact of the existing ones. Specific sub-targets include, among others: - Workshop: Gathering Supply Side and Demand Side Status of Delivery as of May 2012 Rating 7 The December 2011 workshop on EE appliance policies was intended to introduce all stakeholders to the outputs and recommendations of consultant reports on EE policies, market monitoring systems, and testing and conformity assessment infrastructure. -Workshop was attended by MoENR (supply side) and MoSIT (demand side) and other relevant stakeholders -Final workshop on EE appliance policies was designed to obtain consensus amongst all stakeholders on next steps - an assessment report combining a GHG emission reduction and cost benefit analysis; - review of the existing EE appliance program; - review of the new regulations proposed under the EU EcoDesign Directive and acceleration of their transposition in Turkey, including new S&L requirements for TV sets not subject to any S&L schemes yet in Turkey; -Draft assessment report on GHG emission reductions and cost benefit analysis is being reviewed by stakeholders. Final version to be available in June Review of existing EE appliance program was included in the draft assessment report on GHG emission reductions -Commenced training of MoSIT field inspectors in May Target is for a total of 200 inspectors to be trained by late June 2012 Mid-Term Evaluation Mission 12 July 2012

23 Table 1: Project Progress Observed in May 2012 (against January 2011 log-frame from Inception Report) Project Strategy (taken from Prodoc) Measurable Indicators from Prodoc Log-Frame End of Project (EOP) Target - finalized training curricula and modules/ materials; Status of Delivery as of May 2012 Rating 7 -training cirricula and modules completed in advance of MoSIT training sessions - delivered training on adopted policies (At least 5 trainings for 20 participants per training); -Completed training of MoSIT headquarters personnel for implementing EcoDesign and energy labeling regulations in April delivered training on eco-design (at least 5 trainings for 20 participants per training); -Completed training of MoSIT headquarters personnel for implementing EcoDesign and energy labeling regulations in April participation in international and national workshops, meetings and study tours (at least 10 technical persons per year) -Study tour to UK on best international S&L practices was completed in November 2011 with 3 government officers. Output 1.2: A structured market monitoring system Availability of required data Regularly updated data on annual sale of different appliances per energy classes available for public use (with finalized market monitoring methodology and established system with Association of Manufacturers) Market monitoring system has been established with 2011 data from TURKBESD, ISKID and TESID Output 1.3: Agreements with the private sector on the Status of complementary promotional At least 2 consultation workshops and concluded agreements Preliminary informal discussions have taken place between the Project and Arçelik on strategies for incentive Mid-Term Evaluation Mission 13 July 2012

24 Table 1: Project Progress Observed in May 2012 (against January 2011 log-frame from Inception Report) Project Strategy (taken from Prodoc) implementation of voluntary agreements and/or specific promotional campaigns and incentives schemes, for instance, for the accelerated replacement of old inefficient appliances Measurable Indicators from Prodoc Log-Frame measures End of Project (EOP) Target for specific promotional campaigns and/or incentive schemes for at least two appliances. Status of Delivery as of May 2012 Rating 7 schemes and promotion of the phase-out of old appliances. Outcome 2: A structured enforcement and verification program with adequately trained staff and other resources The rate of compliance checked by random samples taken from the market and random visits to the retail stores Over 90% compliance of the random product samples and visits to the retail stores. Initial product testing in 2012 will be outsourced to foreign labs until the investment plan for TSE is implemented (completion of TSE investment expected in 2013 or 2014) PMSP was prepared and will help measure rate of compliance of market place to the desired 90% Output 2.1: A finalized proposal for consolidating the compliance checking and enforcement scheme for products and retailers including testing of products under energy labeling regulations Status of the proposal Finalized proposal for consolidating compliance checking and enforcement scheme both for products and the retailers including energy efficiency aspects (addressing also the required legal amendments to effectively follow-up non-compliance) Proposal was finalized and agreed by MoSIT in April 2012 Output 2.2: Agreed and upgraded procedures and Status of the agreement Agreed and upgraded procedures and organizational Agreement completed with approval by MoSIT in April 2012 Mid-Term Evaluation Mission 14 July 2012

25 Table 1: Project Progress Observed in May 2012 (against January 2011 log-frame from Inception Report) Project Strategy (taken from Prodoc) organizational arrangements for testing of products regarding energy efficiency performance Measurable Indicators from Prodoc Log-Frame End of Project (EOP) Target arrangements for testing of products regarding energy efficiency performance Status of Delivery as of May 2012 Rating 7 Output 2.3: Testing the agreed compliance checking and enforcement schemes for all targeted (6) appliances in selected locations Status of the pilot project The agreed programs, procedures and organizational arrangements tested for all targeted (6) appliances before their broader adoption Proactive Market Surveillance Plan (PSMP) was completed A pilot project to test the PSMP was agreed to by MoSIT in April 2012 Output 2.4: Trained staff of both the selected testing laboratories and MoSIT s branch offices to implement the compliance checking program The amount and type of training provided Specific training courses and/or on-the-job training delivered as per the annual work plans, including training the state inspectors in MoSIT on compliance (estimated 10 trainings for 20 participants per training event) and training on testing of products (estimated 2 trainings for 30 participants per training event) Inspection training is in progress and will be completed in 2Q of 2012 Laboratory training is being planned and is scheduled for completion by Q4 of 2012 Outcome 3: Raised awareness of the end-users and the supply chain and strengthened capacity of the local The priority of different criteria used by the targeted clients in their purchasing decisions Beside the initial purchasing price, energy efficiency and life-cycle costs have become key criteria for purchasing decisions. Mid-Term Evaluation Mission 15 July 2012

26 Table 1: Project Progress Observed in May 2012 (against January 2011 log-frame from Inception Report) Project Strategy (taken from Prodoc) manufacturers to develop and implement specific promotional activities to enhance the sale of energy efficient appliances. Measurable Indicators from Prodoc Log-Frame End of Project (EOP) Target Status of Delivery as of May 2012 Rating 7 Output 3.1: Completed surveys to assess the level of awareness and key drivers of the consumers for the purchase of different products in prior and after the campaign Status of surveys Completed consumer surveys with at least 1500 questionnaires per survey. First survey completed in May 2012 Output 3.2: Joint marketing campaigns with the manufacturers and retail chains (with related material for advertising and instore use) highlighting the energy efficiency aspects and the lifecycle costs approach. Output 3.3: A web site to support consumer s choice with test results and other product information, pricing, easy to use calculation tools etc. Impact of the content of the website in consumers purchasing decisions Impact of the content of the website in consumers purchasing decisions Delivery of joint marketing campaigns with the manufacturers and retail chains highlighting EE aspects and life-cycle cost approach, including as applicable, booklets, billboards, newspaper advertisements, TV spots, fliers, internet Over 20% of the interviewed consumers in stores considering the purchase of a new appliance are aware of and have found the content of the website useful A public awareness raising strategy was prepared in November 2011 and will be revised based on results of the first consumer survey Completed and uploaded in May 2012 Mid-Term Evaluation Mission 16 July 2012

27 Table 1: Project Progress Observed in May 2012 (against January 2011 log-frame from Inception Report) Project Strategy (taken from Prodoc) with an emphasis on energy efficiency Measurable Indicators from Prodoc Log-Frame End of Project (EOP) Target Status of Delivery as of May 2012 Rating 7 Output 3.4: Trained sales staff in the retail chain (complemented, as applicable, by specific incentives such as premiums for the sales personnel for the sale of EE products) to market the products on the basis of their energy performance and related life-cycle costs beside other characteristics Emphasis on EE aspects in the marketing strategy of the retail chain. As applicable, disbursement rate of the incentives for the sales personnel to market EE products. Energy efficiency and life-cycle cost reduction aspects highlighted in the marketing strategy of the retail chain Project has recruited local consultant for training staff, and draft contents of training have been prepared. Training documents in progress and to be finalized with project partners in late June 2012 Training (2-day workshop) is planned for September 2012 Output 3.5: Specific promotional campaigns to expedite phase-out of old inefficient appliances, including, as applicable, specific financial incentives and/or utility (DSM) driven delivery and financing models. Status and the delivery rate of the campaigns Reaching at least 50% of the stated target of the campaigns, as measured by the delivery rate of the promotional measure used. Preliminary informal discussions have taken place between the Project and Arçelik on strategies for incentive schemes and promotion of the phase-out of old appliances. Outcome 4: Institutionalization of the support provided by the project, including monitoring, learning, adaptive The status of recommendations contributing to institutional sustainability. Project recommendations to ensure institutional sustainability adopted and implemented. Mid-Term Evaluation Mission 17 July 2012

28 Table 1: Project Progress Observed in May 2012 (against January 2011 log-frame from Inception Report) Project Strategy (taken from Prodoc) feedback and evaluation. Measurable Indicators from Prodoc Log-Frame The level of information available for adaptive management and for measuring the impact of the project End of Project (EOP) Target Adequate information available for adaptive management and measuring the impact. Status of Delivery as of May 2012 Rating 7 Output 4.1: An updated baseline study, against which the impact of the project can be measured Status of the report. An updated baseline study finalized. Completed in April 2012 Output 4.2: Energy efficiency aspects increasingly included into the curricula of relevant educational institutions. The level of inclusion of appliance energy efficiency aspects into the curricula of relevant educational institutions. Appliance energy efficiency aspects increasingly included into the curricula of the relevant educational institutions, with a specific course on appliance energy efficiency in at least one university. Local consultant to be hired in August 2012 to review local curricula and to develop proposals to increasingly include EE appliance aspects into educational institute curricula Local consultant will also to develop a model in Q4 of 2012 for generating research projects in EE in universities Output 4.3: Further elaboration of the possible financial support mechanisms to accelerate the market shift towards The type of financing available for covering the incremental investment costs of energy efficient Research studies related to the topic of the project started and completed in Turkish universities Identified or established financial support mechanisms continue to promote the purchase of energy efficient appliances at and after See Output 1.3 Mid-Term Evaluation Mission 18 July 2012

29 Table 1: Project Progress Observed in May 2012 (against January 2011 log-frame from Inception Report) Project Strategy (taken from Prodoc) more energy efficient appliances, including, as applicable, carbon financing Measurable Indicators from Prodoc Log-Frame appliances End of Project (EOP) Target the end of the project. Organized stakeholder meetings to discuss the possible financial instruments and mechanisms (at least 5 meetings with banks and other financial institutions) Status of Delivery as of May 2012 Rating 7 Output 4.4: Final project report consolidating results and lesson learnt from implementation of the different project components and recommendations for the required next steps Status of the final report Final project report consolidating the results and lesson learnt from the implementation of the project. To be done at the conclusion of the project Output 4.5: Project mid-term and final evaluations and other required reviews Status of the midterm and final evaluation Finalized mid-term and final evaluations Mid-Term evaluation completed and the final evaluation to be completed by the end of Project Mid-Term Evaluation Mission 19 July 2012

30 To date, the Project has had an indirect impact on GHG reductions through its activities on strengthening the policy and enabling environment for appliance EE compliance testing. Collection of data for baseline energy consumption and EE appliance sales commenced in January 2011, culminating in the delivery of a market database system complete with estimates of cumulative 2011 energy consumption, and estimates of GHG emission reductions to 2011 and A summary of the calculations is contained in Appendix D. For the GHG reductions calculation, sales data for most of the product groups were obtained from manufacturers, importers and their trade associations, such as TURKBESD and ISKID. Since these sources do not cover 100% of domestic sales, some correction factors were included in the calculations to estimate the anticipated total domestic sale from the obtained (partial) sales statistics of the monitored product groups. A summary of estimated CO 2 reductions to date is provided in Table 2. Reductions were reported for appliances where sales data was available for appliances complying with recently developed energy labels such as refrigerators, freezers and washing machines. Reductions could not be reported for other appliances due to energy labels being recently adopted (dishwashers), or under development (stove ovens). In the case of air conditioners, no energy label has yet been adopted due to the need to further disaggregate the appliance type into cooling and heating/cooling types. For televisions, TURKBESD has not collected sufficient sales data of labeled televisions. The estimated GHG reduction by the end of the Project in 2013 is anticipated to be 3.0 million tonnes CO 2eq, considerably more than the original 1.7 million tonne CO 2eq target of the Project. This is due to the accelerated adoption of EU regulations and an updated sales forecast (indicating greater than originally anticipated sales) of white goods sold. Table 2: Summary of CO 2 Reductions from the Project Direct emission reduction 8, t CO2 0 Direct post-project emission reduction 9, t CO2 0 Indirect emission reduction 10, t CO2 Refrigerators-Freezers 290,341 Washing Machines 199,630 Dish Washers 0 Stove Ovens 0 Air Conditioners 0 Televisions 0 Indirect emission reduction, t CO2 489,971 TOTAL EMISSION REDUCTIONS DUE TO UNDP-GEF PROJECT, t CO2 489, These are from pilot projects and investments leveraged during the supervised implementation of the Project. Post-project emission reductions are related to setting financial mechanisms (e.g., revolving funds) that continue operating after the end of the project. In the case of the Project, no such financial mechanisms are planned Indirect emission reductions are from raised awareness, policies or improved regulations generated by Project activities to the end of 2011 Mid-Term Evaluation Mission 20 July 2012

31 2.2 Project Design and Relevance Project Relevance and Country Drivenness This Project has strong relevance to a number of national strategies, plans and programmes including: The Ninth Development Plan of Turkey ( ), which stresses energy efficiency as a means of minimizing environmental damage and satisfying energy demands (as stated under the Section 7.1.5: Development of Energy and Transport Infrastructure ) 11 ; The Medium Term Programme of Turkey, which makes reference to the Energy Efficiency Law and the Plan s support for increasing supporting facilities in the framework of the Law 12 ; The Industrial Strategy Document of Turkey, which makes direct reference in Para. 235 (section 5.6.2) to the priority of conducting studies on energy efficiency of electrical appliances 13 ; The Energy Efficiency Strategy Paper , which outlines the government s strategic measures to reduce energy consumption through a number of strategic programmes that include Market transformation of energy efficient products under the SP-03 programme Project Design and Implementation Approach The Project design and implementation approach are strategic in focusing on 6 appliances that together comprise over 75% of energy consumed by all appliances in Turkey. By focusing on 6 selected appliances, efforts to transform the market for these appliances will lead to tangible emission reductions. In considering the progress of the Project to date and extrapolating what can be achieved by the end of the Project, the Project design is considered appropriate to support a market transformation to energy efficient household appliances. Progress is being made in the strengthening of Turkish appliance EE legislation through harmonization of EU policies, the strengthening of conformance testing systems, and development of a systematic approach to market surveillance that will provide feedback on the progress of the market transformation. Much of the supporting legislation and monitoring tools are being prepared at this time for use in the second half of the Project. Implementation approaches to the Project have been designed, shared with and accepted by the main relevant government agencies, retailers and consumer groups, including the DGRE (under the MoENR), MoSIT, TURKBESD and Arçelik. Other stakeholders, such as MoEU, MoD and MoF, have attended the PSC meetings and various workshops. The result of this approach has been strong support from all stakeholders. One implementation issue concerns the start-up of Output 3.5: Specific promotional campaigns to expedite phase-out of old inefficient appliances, including, as applicable, specific Mid-Term Evaluation Mission 21 July 2012

32 financial incentives and/or utility (DSM) driven delivery and financing models. While EU countries have experience at implementing such schemes, they are not as prevalent today due to the economic problems of EU countries. This should not discourage Project personnel in formulating financial incentives for the renewal of household appliances. The purpose of the incentive scheme is to catalyze and accelerate the phase-out of inefficient appliances, something that is no longer required in EU countries. In conclusion, the Project design and implementation approach have been appropriate, laying the foundation for its successful conclusion within 18 months or potentially longer if justified. 2.3 Project Implementation Arrangements Stakeholder Involvement, Linkages to Project and Other Interventions in Sector The principal Project activities are focused on building the capacity of MoSIT and, to a lesser extent, DGRE (under MoNRE), consumer watchdog groups and private sector retailers. MoSIT is executing much of the market transformation related to upgrading of testing laboratories and enhancing the capacity of its inspectors to enforce appliance EE regulations and standards. Moreover, the Project has facilitated MoSIT linkages with TURKBESD and Arçelik to promote EE appliance sales and the collection of sales data, as well as with the Turkish Standards Institute (TSE) on the upgrading of TSE laboratories to implement a compliance testing programme for EE appliances covered under this Project. All important stakeholders are well engaged with the Project activities. Official communication between the Project and other Turkish government stakeholders (such as MoD, MoEU, MFA and the Secretariat General for EU Affairs) has been mainly through the Project Director, who serves as the Chairman of the Energy Efficiency Coordination Board (EECB) as well as the Director General of DGRE. The Director General of DGRE also coordinates external assistance efforts in the energy efficiency arena in Turkey, including knowledge and resources from: The UNDP-GEF project on industrial energy efficiency; The UNDP-GEF project on energy efficiency in buildings; EU Commission through the IPA (Instrument for Pre-Accession) that is providing an estimated USD 1.0 million technical assistance packages to the Turkish government on supporting ESCOs and energy efficiency in buildings; USAID project on energy efficiency in industrial zones; The recently completed Netherlands Ministry of Economic Affairs technical assistance to MoENR on implementing voluntary agreements on energy efficiency 15 ; Joint EU-World Bank projects on EE (including EE retrofits in buildings) that commenced in late 2011, and a larger project to commence in 2013 on facilitating private sector involvement in EE initiatives for buildings Management, Monitoring and Evaluation, Identification and Management of Risk Management and M&E of the Project have been adequate. This verdict is based on a review of the Project PIRs and associated documentation that provides a clear picture of Project accomplishments and delays, risks and follow-up actions to mitigate risks. One reason for the 15 Mid-Term Evaluation Mission 22 July 2012

33 well-prepared reports is that the CO employs a dedicated M&E officer who assists the Project in the proper preparation of APRs and PIRs. 2.4 Project Budget and Cost Effectiveness Table 4 provides an overview of expenditures of the GEF Project budget of USD 2.71 million from March 2010 to May 31, As of May 31, 2012, USD 425,567, or close to 16% of the GEF-funded Project budget, has been expended. USD 2.28 million remains in the Project budget for technical assistance activities. Component Table 4: Project Budget and Expenditures (in USD) Budget (from Inception Report) 2010 (from Mar 1) (as of May 31) Remaining 1. Enhanced institutional capacities 350,000 19, , , , A structured enforcement and 1,064, ,526 30,232 1,010,242 verification programme 3. Raised awareness / strengthened capacity 820,000 1,050 14,185 28, , Institutionalizing the project support 276, ,689 14, ,060 Project Management 200,000 21,840 18,903 20, ,864 TOTAL 2,710,000 42, , ,645 2,284,433 The cost effectiveness of the Project has been adequate. Although only 16% of the budget has been expended by the mid-point of the Project, this should not be interpreted as being indicative of the progress of the Project considering the large expenditures that are planned in the near-term: Large training sessions planned for over 200 MoSIT field inspectors; Laboratory training sessions scheduled for Q are being planned; Planning for training of retail store sales staff in September 2012; and Completion or substantial completion of all international consultant assignments. The value of these activities would increase the current budget expenditure to over 40%. Moreover, completion of these activities will expend a substantial proportion of the budget by the end of the Project. This will depend to a large extent on how efficiently activities to achieve Outcomes 3 and 4 are implemented. Financial and procurement procedures are in line with UNDP Standard Operating Procedures, ensuring that best value for money is obtained. Additionally, adaptive management has been promoted through careful budget revisions, monitored by the RTA, CO programme team and both Project Advisory and Project Boards. Mid-Term Evaluation Mission 23 July 2012

34 Confirmed co-financing leveraged by the Project is USD 2,247,000. To date, USD 1,608,620 has been co-financed by Project partners to June 2012, the details of which are provided in Table 5. This is 55% of the USD 2,946,600 co-financing cited in the ProDoc. The Project is in position to mobilize more than the USD 2,946,600 co-financing notably from stakeholders participating in Outcomes 3 and 4 (Arcelik, TURKBESD and other industrial associations). Partner Agency Co-Financing Amount to June 2012 (USD) DGRE/MoENR 113,520 Arçelik 1,420,000 TURKBESD 34,100 MoSIT 41, Evaluation of Project Table 5: Details of Project Co-Financing Activities Attended various meetings, workshops and training programmes, delivered presentations, provided project office, meeting rooms and training rooms for workshops and training programmes, participated study tours. Provided meeting rooms for various meetings, participated in various meetings and workshops. Designed and implemented advertisements in visual and printed media to promote energy efficient products also serving the objective of the project. Provided meeting rooms for stakeholder meetings, participated in various meetings and workshops where they ensured participation of their members and delivered presentations. Organized press release. Attended various meetings, workshops and training programmes with their HQ and provincial office staff, delivered presentations, provided meeting rooms, participated study tours. Table 4 provides an evaluation of the current outcomes of each Project output. Each output was evaluated against individual criteria of: Relevance the extent to which the activity is suited to local and national development priorities and organizational policies, including changes over time. Effectiveness the extent to which an objective has been achieved or how likely it is to be achieved. Efficiency the extent to which results have been delivered with the least costly resources possible. Results/impacts the positive and negative, and foreseen and unforeseen, changes to and effects produced by a development intervention. In GEF terms, results include direct project outputs, short-to medium term outcomes, and longer-term impact including global environmental benefits, replication effects and other, local effects. Sustainability the likely ability of an intervention to continue to deliver benefits for an extended period of time after completion. Projects need to be environmentally as well as financially and socially sustainable. The Project outputs were rated based on the following scale: Highly Satisfactory (HS): The project has no shortcomings in the achievement of its objectives; Mid-Term Evaluation Mission 24 July 2012

35 Satisfactory (S): The project has minor shortcomings in the achievement of its objectives; Moderately Satisfactory (MS): The project has moderate shortcomings in the achievement of its objectives; Moderately Unsatisfactory (MU): The project has significant shortcomings in the achievement of its objectives; Unsatisfactory (U) The project has major shortcomings in the achievement of its objectives; Highly Unsatisfactory (HU): The project has severe shortcomings in the achievement of its objectives. The overall rating of the Project is Satisfactory (S), based mainly on: Satisfactory progress on all outcomes at the mid-point of the Project with few or no delays from the original schedule. This includes timely and effective delivery of the transposition of EU legislation, and knowledge transfers to MoSIT officers on compliance checking and enforcement; Table 4: Summary Evaluation of Project Project Strategy Relevance Efficiency Outcome 1: Enhanced institutional capacities in Turkey to develop and implement effective appliance EE policies Outcome 2: A structured enforcement and verification programme with adequately trained staff and other resources Outcome 3: Raised awareness of the end-users and the supply chain and strengthened capacity of the local manufacturers to develop and implement specific promotional activities to enhance the sale of energy efficient appliances Outcome 4: Institutionalization of the support provided by the project, including monitoring, learning, adaptive Effectiveness Overall Rating S MS 16 S S S S S S S S Unable to rate 17 Unable to rate 18 Unable to rate Unable to rate Unable to rate Unable to rate feedback and evaluation Monitoring and Evaluation S S S S Overall Rating S S S S 16 Project efforts to transpose EU legislation did not commence until January 2011, 9 months after the March 2010 start of the Project 17 Due to insufficient completion of activities. 18 Due to insufficient completion of activities Mid-Term Evaluation Mission 25 July 2012

36 Current preparations for numerous training workshops in 2012 that will strengthen an excellent foundation for market transformation for the remaining period of the Project. Current preparations include sales and inspector training on EE issues, pilot testing of the PMSP, and implementation of an awareness programme on EE appliances; Less than 16% of resources spent to date at the mid-term of the Project, translating into resources being available for capacity building activities for remainder of the Project. 2.6 Sustainability and Replicability Sustainability In assessing the sustainability of the project, we asked how likely will Project outcomes (from the revised log-frame of the January 2011 Inception Report) be sustained after termination of the Project. Sustainability of these objectives was evaluated in the context of financial resources, socio-political risks, institutional framework, governance and environmental factors, using a simple ranking scheme: Likely (L): very likely to continue and resources in place; Moderately Likely (ML): model is viable, but funding or resources may not be in place; Moderately Unlikely (MU): model is not viable or needs changing; and/or resources not in place; and Unlikely (U): model is not viable and resources are not in place The evaluation for sustainability is shown on Table 5. It is important to note that the index is intended simply to facilitate an assessment of future sustainability and is not a rating of project management and consultants. Instead, it is a rating of the project design and viability going forward, including availability of budget and resources for continuation. Project sustainability rating is Likely (L) due to: o o The strong engagement of all stakeholders interviewed on the Project; A strong foundation built for market transformation that includes: EE being a high government priority; the transposition of EU Energy Labeling and EcoDesign regulations into Turkish; the Project plan for market surveillance system being accepted by government; strong commitments of government to adopt a Project-designed market monitoring system database; strong commitments for field testing of equipment; and excellent attendance at training sessions for field inspectors. Mid-Term Evaluation Mission 26 July 2012

37 Outcome Outcome 1: Enhanced institutional capacities in Turkey to develop and implement effective appliance EE policies. This includes: new legal and regulatory provisions for compliance checking based on best international practices trained staff to develop and implement EE policies such as review of new EU regulations for transposition, management of a market monitoring system, and working with the private sector on promotional and incentive schemes Outcome 2: A structured enforcement and verification programme with adequately trained staff and other resources. This includes: Design of a compliance checking and enforcement scheme for EE appliances and retailers Training of inspection staff to implement the compliance checking and enforcement scheme Outcome 3: Raised awareness of the end-users and the supply chain and strengthened capacity of the local manufacturers to develop and implement specific promotional activities to enhance the sale of energy efficient appliances. This includes: Delivery of joint marketing campaigns; Development of web-based information platforms to promote EE appliances; Trained retail sales staff to improve sales of EE appliances Table 5: Assessment of Sustainability for Objectives Assessment of Sustainability Financial Resources: Financial resources are available to government agencies to support and implement appliance EE policies based on the market transformation of appliances being a strategic priority of the Energy Efficiency Strategy Paper. Private sector resources will also be available; Socio-Political Risks: Manufacturers of EE appliances are the drivers of EE standards that make Turkish appliances competitive. The appliance manufacturing sector is also a major source of employment for Turkey, further reducing this risk; Institutional Framework and Governance: The market transformation of EE appliances is a strategic priority of the Energy Efficiency Strategy Paper ( ); Environmental Factors: Reduced energy consumption and GHG emissions are a consequence of market transformation activities of the Project. Overall Rating Financial Resources: Financial resources are available to government agencies and the manufacturers to support and implement the compliance checking and enforcement scheme and testing programmes as these activities will assist in transforming the market for EE products and help drive sales of Turkish manufactured goods; Socio-Political Risks: Manufacturers of EE appliances are drivers of the Turkish economy. As such, the socio-political risks are low that these compliance schemes and testing programmes will be resisted by any stakeholder; Institutional Framework and Governance: The market transformation of EE appliances is a strategic priority of the Energy Efficiency Strategy Paper ( ). These actions only strengthen the transformation of the market towards EE appliances; Environmental Factors: Reduced energy consumption and GHG emissions are a consequence of market transformation activities of the Project. Overall Rating Financial Resources: Private sector financial resources will be available from advertising budgets to enhance sales of EE appliances; Socio-Political Risks: Manufacturers of EE appliances are significant drivers of the Turkish economy. As such, the socio-political risks are low that these awareness programs will not be implemented by the private sector and government agencies; Institutional Framework and Governance: The market transformation of EE appliances is a strategic priority of the Energy Efficiency Strategy Paper ( ). These actions only strengthen the transformation of the market towards EE appliances; Environmental Factors: Reduced energy consumption and GHG emissions are a consequence of market transformation activities of the Project. Overall Rating Dimensions of Sustainability L L L L L L L L L L L L L L L Mid-Term Evaluation Mission 27 July 2012

38 Outcome Outcome 4: Institutionalization of support provided by the Project including monitoring, learning, adaptive feedback and evaluation. This includes: Updating of baseline studies; Inclusion of appliance energy efficiency aspects into curricula of relevant educational institutes; Development of financial support mechanisms to promote the sales of EE appliances after the completion of the Project; Workshops and seminars to share lessons learnt from the implementation of the Project Table 5: Assessment of Sustainability for Objectives Assessment of Sustainability Financial Resources: There will be sufficient financial resources from the EE appliance manufacturers to institutionalize Project support as this will improve EE appliance sales; Socio-Political Risks: Manufacturers of EE appliances are significant drivers of the Turkish economy. As such, the socio-political risks are low that there will not be any institutionalized support for improving S&L initiatives for Turkish manufactured products; Institutional Framework and Governance: These activities help transform the market for EE appliances which is a strategic priority of the Energy Efficiency Strategy Paper ( ; Environmental Factors: Reduced energy consumption and GHG emissions are a consequence of market transformation activities of the Project. Overall Rating Dimensions of Sustainability L L L L L Replicability Project replicability is likely due to: MoSIT having dedicated personnel for transposition of EU regulations in its Directorate General for Safety and Audit of Industrial Products. MoSIT personnel demonstrated that transposition of EU regulations will continue beyond the project for other regulations, which will help Turkey meet conditions for EU accession and strengthen its market share for manufactured goods in the EU; The training of field inspectors that has been well received, as shown to the MTE mission during the May 29 field inspectors workshop in Ankara; Strong engagement of the private sector on project activities, including Arçelik (as a project partner) and Vestel (which has had discussions with the Gold Standard Foundation on the possibility of utilizing carbon finance to accelerate its sales of their EE appliances). Mid-Term Evaluation Mission 28 July 2012

39 3. CONCLUSIONS AND RECOMMENDATIONS 3.1 Conclusions The Project has laid an excellent foundation for market transformation based on the progress of the EU legislation transposition, the design and implementation of a market surveillance system, the training of field inspectors to check compliance and enforce standards, and the progress of work achieved thus far in setting up of testing laboratories to implement the compliance checking programmes. One issue that needs attention during the remainder of the Project is the quality of energy consumption data received from industry associations: Industry associations are only reporting appliances sold by energy classes. To obtain a more accurate energy consumption figure for appliances, sales figures should be reported as a weighted average of each appliance (each appliance has a specific energy consumption figure). Such figures are not disclosed due to manufacturer s fears of revealing information that would affect their competitiveness with other name brands; The Regulation on Increasing Efficiency and Use of Energy and Energy Resources obliges the suppliers, manufacturers and importers of household appliances to submit the annual sales by energy classes. There is, however, no provision either in this regulation or elsewhere obliging MoSIT to report sales data, energy consumption and GHG emissions by household appliances; With MoSIT not generating such data, the responsibility falls on the suppliers to submit such data. While the Project proposed an amendment obliging suppliers to submit sales-weighted average energy consumption figures, receipt of such data to be entered in the Market Monitoring Tool (as developed under the Project) will be a challenge for the remaining period of the Project; The importance of resolving this issue cannot be understated as it will enable MoSIT to report GHG emissions to MoENR (on energy consumption data) and MoEU (which is responsible for reporting GHG emissions to the UNFCCC on behalf of the government). Despite the progress achieved thus far, there is a strong likelihood that the Project completion date may need to extend beyond December 31, Activities contributing to this delay may include: Completion of activities for Outcome 3: Design of promotional campaigns and financial incentives to accelerate EE appliance market transformation. Time will be required to formulate and design a promotional campaign to phase-out old inefficient appliances with financial incentives. One significant issue is how to manage the appliance waste given there is no integrated waste management strategy for old inefficient appliances. With the high cost of metal, a recycling programme for waste metal from old appliances will be a necessity for such a programme to work. Moreover, it is likely that more than 18 months will be required to obtain acceptance of such re-cycling measures with the manufacturer and relevant government agencies; Piloting the PMSP and the arrangements to test equipment (as related to Output 2.4). The TSE will require more than 18 months to upgrade its testing laboratories for air conditioners: Mid-Term Evaluation Mission 29 July 2012

40 o o o First, it will require clarification on specifications of air conditioners to be tested. TSE estimates that MoSIT will require another 6 to 12 months to decide on the required air conditioner tests; Second, it will need to design and estimate costs for an upgraded laboratory with new equipment. TSE estimates that an additional 6 to 8 months are required to design the laboratory upgrading (for testing air conditioners) and to obtain the necessary financing for the facility; and Third, it will require approximately 12 months to physically upgrade the laboratory. During the interim period, MoSIT will be outsourcing equipment testing via TSE to other laboratories in or outside Turkey, such as the facilities operated by Arçelik. However, the long-term target of having the TSE as the main facility for testing equipment is not likely to be achieved until mid-2014, beyond the scheduled Project completion date; Further changes to government institutions. This is a risk beyond the control of the Project that can be partially mitigated by the Project Director, who will be monitoring planned changes to the relevant institutions. 3.2 Recommendations To improve the likelihood of the EE Appliances Project achieving its intended outcomes, the following recommendations are provided: Recommendation 1: Improve quality of energy and GHG data received from industry associations and other sources: MoSIT, with guidance from MoENR and MoEU, will need to provide specific direction to industry associations and manufacturers on reporting sales on the basis of weighted average energy consumption for each product group. This will involve using sensitive product sales data which does not need to be disclosed during the reporting process. Such an effort would be in parallel with MoEU s new regulation of April 2012 obliging certain facilities to setup annual monitor, report and verify (MRV) of greenhouse gas (GHG) emissions by January The Government has designated TURKAK 19 (Turkey s sole accreditation agency) as its third party to approve a facility s MRV plans; Private market research companies may need to be employed to improve data collection and quality of data as a backup plan in the event there are reporting compliance problems with the various manufacturers and industry association representatives; With MoEU being responsible for reporting GHG emissions to the UNFCCC, it also needs to be involved with quality assurance of GHG data reported to it in close collaboration with TURKAK. TURKAK is now designing the installation-level monitoring systems to EU-ETS standards; 19 Mid-Term Evaluation Mission 30 July 2012

41 Recommendation 2: Explore possible financial incentives to accelerate EE appliance market transformation such as: A government subsidy for the purchase of an EE appliance. Such a request needs to be backed by a good business case for the government, and may require the assistance of the EE Appliances Project, as directed by DGRE and the appliance manufacturers; Linking appliance purchases with the country s voluntary carbon finance markets, with the replacement programme designed with assurances that the old appliances are exchanged for new appliances; and An ESCO business model. This model may not work for household appliances, but could be applied to large industrial-scale installations, such as central air conditioners and heaters. This is likely outside the scope of the Project. Recommendation 3: The Project needs to leave open the possibility of extending the project terminal date to sometime in 2014 to allow the project sufficient time to achieve its objectives. The MTE mission feels that the Project will likely take more than 18 months to complete the following outputs: Set-up of financial incentives (related to Outcome 3). A likely scenario is to have each manufacturer with its own programme linked with a voluntary carbon scheme. This may take more than 18 months to establish, depending on how quickly the manufacturers adopt their sales to the carbon scheme; Build-up of TSE testing laboratories (notably with air conditioners) and the subsequent agreements and arrangements of MoSIT to use TSE on a long-term basis for testing of the 6 groups of appliances covered under the project. TSE will likely need more than 18 months to develop air conditioner testing equipment, including establishment of the plan of which air conditioners to test (split, wall units and for the type of climatic zone), procurement of testing equipment, installation and commissioning of equipment. 3.3 Lessons Learned Two valuable lessons from the MTEEA Project are as follows: GEF is a window for recipient countries to international expertise that can provide the best international practices. If used properly, GEF international consultancy contracts can provide an excellent foundation for project implementation; Adaptive management is important to improve the likelihood of a project achieving its intended outcomes. The Project, thus far, has demonstrated adaptive management after the transposition of the new EU Directives on Eco-Design and Energy Labeling. At this time, training needs were identified for MoSIT field inspectors in the area of energy efficiency and how to ensure compliance and enforce transposed EU Directives. In response to the high interest shown by government officials, MTEEA project management organized and provided the requested training with the MTE Mission observing that the trainees have embraced the training process. MoSIT demand for the training has increased to the extent that initial sessions are designed to train-thetrainers and to cover topics ahead of schedule such as Management of Market Surveillance Programme. Mid-Term Evaluation Mission 31 July 2012

42 APPENDIX A MISSION TERMS OF REFERENCE TERMS OF REFERENCE FOR MID-TERM EVALUATION OF THE UNDP/GEF Project: PIMS 4014: Market Transformation of Energy Efficient Appliances in Turkey (EE Appliances) Project Title: Vacancy Type: Location: Category: Type of contract: Reporting Language: Application Deadline: Starting Date: Expected duration of Assignment: Duration of Contract: Reference Code: Market Transformation of Energy Efficient Appliances in Turkey (EE Appliances) One (1) External Vacancies Turkey (Ankara and visits to Istanbul) Environment and Sustainable Development (ESD) IC (Individual Contract) English 6 April April man/days throughout the contract validity (non-consecutive) 16 April June 2012 Midterm Evaluation International Consultant (EE APPLIANCES MTE_INTERNATONAL) 1) A. INTRODUCTION - Standard UNDP/GEF M&E requirements This Mid Term Evaluation (MTE) is initiated by the UNDP Turkey as the Implementation Agency for this project and it aims to provide managers (at the Project Implementation Unit, UNDP Turkey Country Office and UNDP- GEF levels) with strategy and policy options for more effectively and efficiently achieving the project s expected Mid-Term Evaluation Mission 32 July 2012