Timor-Leste: CPS 2011 2015 Final Review Validation, Linked Document 7 ACHIEVEMENT OF TECHNICAL ASSISTANCE OUTCOMES Road Transport TA 7698 - Supporting Road Network Development Outcome: The Ministry of Infrastructure and selected development partners (such as Japan International Cooperation Agency and ADB) have improved understanding of the requirement of road network development. There is no discussion in the CPSFR. The TCR rated the TA successful as the intended outcome was achieved and exceeded. Core documents are shared among development partners and common approaches have been adopted for project design, including application of safeguards and geometric standards. Water Supply TA 4869 - Dili Water Supply Performance Improvement Outcome. Improved DNAS organization The TCR rated the TA partly performance in its Dili City water supply business successful. The TA provided the (i) NRW ratio for Dili city reduced from 95% to 40 executing agency with management 50%, citywide, by month 30; systems and skills necessary for (ii) Monthly water sales revenue for Dili city reaches DNSAS to better manage the Dili $20,000, by month 30; water system. It also raised awareness (iii) Operating cost coverage is 40%, citywide, by within the executing agency and the month 30 (early focus on commercial, institutional, implementing agency of areas that formal housing, and communal/bulk customers); needed to be improved or (iv) Procurement contracts let, NRW targets set, strengthened in the near future. and work plans in place for ongoing leak detection However, the design and the success and NRW reduction work in future target zones, by of the TA were dependent on three month 30. assumptions that did not take place, and so the TA did not achieve the expected outcomes. Output 1. Demand management task force and leak detection teams are established and operating effectively as part of DNAS ongoing Dili city water supply operations. (i) NRW ratio for target zones reduced from 95% to 25 30% by month 30; (ii) Demand management task force (DMTF) established with agreed mandate by end of month 1 and retained for ongoing NRW reduction work by month 30; (iii) Three leak detection teams established by month 2 and team members retained by DNAS as permanent staff by month 30. Output 2. Management, HR and reporting skills, and systems are upgraded and integrated with Dili City water supply operations. (i) Monthly management reporting system implemented in DNAS by end of month 6 and incorporated in DNAS mandatory reporting by month 30; (ii) Monthly key project indicators for Dili city WS published and submitted routinely to MoNRMEP by end of month 6 and incorporated in DNAS Not delivered. The DMTF and the leak detection and caretaker teams were not in place by the end of the TA. The consultants were therefore not able to carry out the activities to support the DMTF or to provide training on leak detection (output 1). Delivered. The TA successfully upgraded the management, human resources, and reporting skills of DNSAS. The NRW ratio did not improve and even deteriorated slightly to 96%. Monthly sales revenue was around $20,000 as targeted. NRW ratio was at 96%. Only one leak detection team has been established. Reports were generated but apparently not regularly, as the latest one provided to IEM was in January 2014.
2 Linked Document 7 mandatory reporting by month 30; (iii) Upgraded duty statements for Dili city WS operations staff by month 30. Output 3. Technical staff apply newly acquired skills to update and maintain new network model and drawing record systems. (i) Hydraulic network model covers all of Dili city water supply distribution system (including tertiary level pipes) by month 30; (ii) Computerized water supply record drawings and drawing registration system cover all of Dili city water supply system by month 30; (iii) Service connection record drawing system (GIS) covers 90 100% of customers by month 30. Output 4. Support Services Section staff apply newly acquired meter management skills and systems. (i) Metering level of all commercial, institutional, formal housing, and communal/bulk customers is 90 100% by month 30, citywide. Output 5. Customer Services Section staff apply newly developed customer management, billing and collection skills, systems and policies. (i) Billing of registered customers (metered and unmetered) is 90% 100% by month 30, citywide; (ii) Collection ratio is 60% 80% by month 30, citywide; (iii) Number of registered customers are 14,000 16,000 by month 30; (iv) Unauthorized connections are reduced from 4,000 to less than 500 in target zones by month 30. Output 6. Technical staff apply newly acquired skills in distribution planning and management. Delivered. The TA successfully upgraded the skills of technical staff to update and maintain a new network model and drawing record systems. Delivered. The customer service section of DNSAS received training on meter management but had little opportunity to apply these skills because few meters were in place by the end of the TA. Partially delivered. The lack of opportunities to apply the skills acquired in the trainings was also problematic in the case of outputs 5 and 6. While technical staff received training on distribution planning and management, they were not able to apply these skills because the rehabilitation of the subzones by the DUWSSP had not commenced by the end of the TA. Similarly, the customer service section received support and training but did not apply their new skills on billing and collection systems because (i) tariffs had to be introduced in the Dili water system, and (ii) DNSAS had to have an independent financial status to implement its own billing system, both of which have not taken place. (i) Planning of leak detection work completed in target zones by month 4 and in future target zones by month 30; (ii) Subzone caretakers recruited, trained, and operating in target zones: maximum one manager per 1,000 customers (in subzone) by month 30. TA 7981 - Strengthening Water Sector Management and Service Delivery Outcome. The outcome of the TA is that MOI and NDWSS will implement a tested urban water sector institutional reform program. At least three district capitals have made budget provisions to implement the tested urban water sector institutional reform program by 2015. Output 1. Adoption of a draft urban water sector institutional reform program. 100% of key water sector stakeholders (MOI, NDWSS, and 13 DDWSS offices) agree by 2014 to test the draft urban water sector institutional The DNSAS staff were trained but were not able to apply their newly acquired skills. The DNSA staff were trained but were not able to apply their newly acquired skills.
Achievement of Technical Assistance Outcomes 3 reform program. Output 2. NDWSS, Pante Macasar DDWSS, and Manatuto DDWSS successfully pilot proposed urban water sector institutional reform program. Percentage of bills collected by Manatuto DDWSS and Pante Macasar DDWSS increases from 0% in 2011 to 80% by 2014. 100% of customer complaints are resolved satisfactorily within 2 weeks by 2014 TA 8064 - Second District Capitals Water Supply SSTA. The project is expected to reduce the incidence of water-related diseases and reduce the workload, particularly of women, in Baucau and additional district capitals. The TA will: (i) develop design requirements for the three water supply systems proposed under the project and a sector investment plan for the remaining capitals; (ii) prepare technical and economic feasibility studies; (iii) ensure compliance with ADB s safeguards policy and develop a Consultation Plan and Gender Strategy; (iv) prepare a realistic estimate of project cost and an implementation timeframe; (v) develop suitable contract packages and prepare a procurement plan; and (vi) assess implementing agencies and community counterparts, and build their capacity to minimize procurement, operations and maintenance, and financial management risks. MULTISECTOR TA 4942 - Infrastructure Project Management Outcome. Transport, communications, urban municipal, power, and water supply and sanitation infrastructure assets are created and upgraded in line with medium-term National Development Plan targets. By 2011, MoPW, MoTC, MoNRMEP achieve annual targets for their respective sector investment programs (SIPs) for transport; housing and urban development; communications; power; water supply and sanitation. Output 1. MoPW, MoTC, and MoNRMEP have completed annually programmed procurement of consulting services, goods, and works contracts, in a transparent, fair, and efficient manner. For MoPW, MoTC, MoNRMEP, by end of the TA, 80% of the Ministry Annual Procurement Action Plan is achieved on time. Project officer's update: Work is underway. The water supply and sanitation master plans for four district capitals are expected to be produced by the end of 2014. No progress/status updates for Impacts and Outcomes in eops. In 2010, 61% of procurements for capital development projects under MOI's procurement authority (between $250,000 and $1 million) have been completed. In 2010, the draft tendering documents for all projects above $1 million were submitted to Procurement Technical Secretariat by June 2010. However, the procurements were delayed until October, and subsequently returned to MOI for evaluation and procurement. MOI completed the bid Likely to be achieved. Physical targets in all sectors were not being met (see Output 2). No recent updates were given, but it is unlikely that 80% of the annual procurement action plan was achieved on time.
4 Linked Document 7 Output 2. MoPW, MoTC, and MoNRMEP have implemented annually programmed consulting services, goods, and works contracts, in a transparent, fair, and efficient manner. (i) For MoPW, MoTC, MoNRMEP, by end of TA, 70% of capital development items contained in the Ministry Annual Infrastructure Work Plan are completed on time and meet agreed-upon specific quality standards, including budget overrun tolerance; (ii) 80% of projects are adequately supervised against established international quality criteria; (iii) 70% of due payments are made within the time specified in the contract/agreement. Output 3. Infrastructure project management systems and processes, and the competencies of MoPW, MoTC, and MoNRMEP staff have improved For MoPW, MoTC, MoNRMEP, by the end of the TA: (i) All finance and administration professional staff are competent to use Freebalance software; (ii) Systems and processes to prepare ministry annual infrastructure work plans, procurement action plans, and project management monitoring and evaluation are functioning effectively and efficiently; (iii) All tertiary-qualified infrastructure planning and engineering staff have managed at least 4 complete procurement cycles: 50% of contract procurement and contract management activities are successfully undertaken by ministry staff unsupervised; (iv) Project management staff assess projects for crosscutting/social safeguard implications and refer them to specialized agencies as required. evaluation and contract awarding for 47% of projects above $250,000. If capital projects waiting for approval of contract award are considered, the percentage of procurement completion will increase to 94%. (i) Transport (land, sea, and air): 39 projects were in the 2010 budget: one project has been completed, civil works of 12 projects are ongoing, 11 projects have awarded contracts, and 15 projects were cancelled; Communications: three projects were in the 2010 budget; no project has been completed, the civil works of the three projects are ongoing; Power: eight projects were in the 2010 budget: no project has been completed, civil works of five projects are ongoing, one project has awarded contracts, and two projects were cancelled; Water supply and sanitation: all projects are less than $250,000 and are implemented under a separate system (PDD) which the TA is not involved in; (ii) 80% of projects are adequately supervised against established international quality criteria; (iii) 70% of due payments are made within the time specified in the contract/agreement. (i) The contractor of MOF conducts regular training on the use of Freebalance. Only selected MOI staff such as those in National Directorate for Administration and Finance and Director General Corporate Services have access to Freebalance. MOI uses Freebalance for issuance of cash payment vouchers and purchase orders and preparation of reports, i.e., budget execution performance; (ii) The organic law of MOI was approved on 19 November 2010. The system and processes to prepare annual work plans are to be established under the framework of the organic law. The TA consultant assisted MOI in preparing the procurement action plan. A project monitoring system has been developed by the TA consultants; (iii) All contract management activities are done by MOI staff with Physical targets (item i) in all sectors are not being met. Nonphysical targets (items ii and iii) are reportedly being met. Substantially achieved. Most targets achieved. But MOI procurement staff still require close supervision (item iii), and MOI does not have social safeguard policies (item iv).
Achievement of Technical Assistance Outcomes 5 TA 8278 - Infrastructure Management Outcome. Strengthened infrastructure management by the MPW. Achievement of at least half of the 2014 targets and/or indicators of the MPW s long-term strategic plan. Output 1. Results-based, long-term strategic plan for the MPW. (i) Endorsement of the plan by the Council of Ministers by the end of 2013; (ii) Gender-specific targets and/or indicators included in the plan by the end of 2013; (iii) Plan released publicly by the end of 2013; (iv) Progress against the plan is reported publicly in 2014. Output 2. Operational capacity development framework. (i) Endorsement of a capacity development framework by the minister of public works by the end of 2013; (ii) Inclusion of the framework in the endorsed MPW strategic plan by the end of 2013; (iii) Gender-specific targets and/or indicators included in the capacity development framework by the end of 2013; (iv) At least two development partners support new capacity development initiatives identified in the MPW strategic plan, in 2014. Education the support of TA consultants. MOI procurement staff still require close supervision to undertake procurement activities; (iv) Environmental safeguard requirements are followed by MOI project management staff, but no social safeguard policy exists in MOI. TA 7120 - Capacity Building to Strengthen Public Sector Management and Governance Skills, Phase 3 Outcome. Enhanced operations and functional skills of civil servants in service delivery, particularly in the sub-districts. Government performance review. The TCR rated the TA successful. Data needed to assess performance against the impact and outcome statement are unavailable, but the TA has established a methodology that will enable evaluation in due course. Nonetheless, it is already clear that the TA has contributed to an ambitious development phase of INAP that will in due course strengthen the capacity of the civil service. Output 1. Civil servants successfully trained in professional competencies in public administration and governance. INAP conducted 122 courses and trained a total of 3,653 civil servants over 2010 2011 (up from 77 training courses and 1,661 participants over
6 Linked Document 7 (i) 120 district and sub-district civil servants educated and trained in public administration and good governance; (ii) System of accreditation and equivalency established for NIPA courses. Output 2. Institutional capability of NIPA to deliver high quality and demand driven education and training programs on public administration and governance strengthened. (i) Accreditation of NIPA courses by NUTL and/or Ministry of Education; (ii) Increased collaboration between NIPA and NUTL. Output 3. Access to and use of instructional materials and knowledge products on public administration and good governance. 2008 2009). No discussion in TCR. No discussion in TCR. (i) Formal agreement to access and use instructional materials and knowledge products secured. Public Sector Management TA 7401 - Statistical and Macroeconomic Capacity Building Outcome. Enhanced capacity of NDS in managing The TCR rated the TA successful. Timor-Leste's statistical system and enhanced capacity of the National Directorate of Macroeconomy to provide macroeconomic advice and analysis. (i) NDS able to execute its mandatory responsibilities; (ii) Macroeconomic advice provided as required. Output 1. Staff development in managing a national statistical office and English language skills. (i) Twinning arrangement in place; (ii) Lessons learned are applied on-the-job through improvements to existing practices; (iii) Staff in NDS are able to converse and write reports in English, as well as understand statistical documents and manuals that are written in English. Output 2. Statistical business register established at NDS and business activity survey and basic statistical skills and computer software course completed. (i) Statistical business register is used as population frame in sample design and other statistical data collection activities; It was recognized that much of the reference material on statistics is only available in English, and that many statistical conferences are also only conducted in English. The low level of English language skills among NDS staff had been previously identified as a serious impediment to effective technical assistance, including being able to take full advantage of the twinning arrangement with the Australian Bureau of Statistics. However, due to other NDS priorities, the English language training did not proceed. For these reasons combined with the in-kind support provided by ABS 28.5% of the TA funds remained unutilized at closure. A key output was the completion of the 2010 business activity survey, which is a key source of information for the development and monitoring of economic policies. The data will help the government identify and address policy issues concerning the performance of the Timorese business Likely achieved. Certain components and outputs under the TA were not done (output 1 and output 3). The English language training did not proceed. Business activity survey completed.
Achievement of Technical Assistance Outcomes 7 (ii) Staff in NDS are able to process data using computer software and analyze statistics independently; (iii) National accounts include timely and robust business activity survey data. Output 3. Agricultural production survey and basic statistical skills and computer software course completed. (i) NDS processes and stores data using computer software, and develops databases in formats which permit easy use for national income computations; (ii) Staff analyze statistics independently; (iii) National accounts include timely and robust agricultural production survey data. Output 4. Capacity to provide macroeconomic advice and analysis strengthened. (i) Macroeconomic analysis prepared and being utilized. TA 7712 - Strengthening Major Public Investments Outcome. The capacity of MPS (or its successor) is strengthened in project preparation and implementation. (i) By 2012, the business processes for project implementation are followed by MPS (or its successor) for more than 25% of projects. (ii) By 2012, the business process for monitoring and reporting is followed by MPS (or its successor) for more than 25% of projects. Output 1. Business processes are established in MPS (or its successor) for project implementation. By Q4 2011, the business processes for project implementation are approved by MPS (or its successor) and applied in its new projects. Output 2. Monitoring mechanism is established and functioning. environment as a whole, as well as progress related to specific sectors of the economy. The survey also gave the government, and other stakeholders, including the business sector itself, an idea of employment numbers and the size and composition of the revenues, expenses, and profits of enterprises operating in different parts of the economy. Given the usefulness of survey, the NDS is now planning to conduct business activity surveys on an annual basis. The agricultural production survey did not proceed. The NDS completed a population census in 2011, which provided data on rural households. Furthermore, the Ministry of Agriculture and Fisheries was considering an agricultural census in 2011. For these two reasons, the NDS agreed not to undertake the agricultural production survey. The approach taken to the macroeconomic component of the TA was the engagement of a number of newly-graduated economists, supported by an international consultant, to improve the economic capacity and capability of the NDM. A quarterly economic review was produced by this team. At the end of their term, the MOF recruited five of the seven economists, who passed an entrance test, and assigned them to various MOF Directorates. Ongoing, but no progress/status updates for Impacts and Outcomes in eops. The agricultural production survey did not proceed. No Outputs were due in 2011. Outputs were due in 2012.
8 Linked Document 7 (i) By Q2 2012, project data are maintained in the monitoring and evaluation system and are updated on a quarterly basis for at least 50% of the projects of MPS (or its successor). (ii) By Q2 2012, project progress reports are prepared on a quarterly basis for at least 25% of the projects of MPS (or its successor). Output 3. National staff of MPS is capable of leading and monitoring project implementation. By Q2 2012, 85% of trained staff understand and can apply the business process for project formulation, project implementation, and monitoring and evaluation. TA 7741 - Support for Effective Aid Management Outcome. Improved processes and systems within the Ministry of Finance for managing development partner assistance. Public Expenditure and Financial Accountability score on indicator D-2 (financial information provided by development partners for budgeting and reporting on project and program aid) of at least C in 2012 2013 (2010 baseline: score of D+). Output 1. Improved information flows on development partner funded activities. (i) Paris declaration indicators 3, 5a, and 7 are publicly reported in 2012 (baseline: no previous reporting); (ii) At least 75% of the expenditure-based indicators of the government s aid management policy are publicly reported by 2012 (baseline: no previous reporting). Output 2. Linked aid management information and financial management information. Reporting of development partner funded expenditure in the public accounts starts by 2012 (baseline: no previous reporting). The TCR rated the TA successful. The TA has effectively empowered the MOF and DPMU through the establishment of AIMS and ATP, and has met all the targets described in the DMF. The TA has contributed to an improved alignment of aid flows with national development priorities through the establishment of the ATP, which shows the activities of development partners against national priorities and OECD-defined sectors. There is also alignment of information including the geographical locations of activities with ministries and departments. Through reports generated in the open portal, information flows on development partner funded activities have been improved and can be transparently seen by all stakeholders. The 2012 and 2013 annual development partners meetings and the 2013 budget used reports directly generated from the AIMS ATP. TA 8256 - Preparing for Regional Economic Integration Outcome. Improved ability by Timor-Leste to carry out the obligations of membership of REI initiatives. Confirmation by the ASEAN Secretariat that Timor- Leste has met at least half the country-specific actions of the ASEAN Economic Community Blueprint by the end of 2015 (2012 baseline: 0%). (i) Lower tariffs on imports; (ii) Reduction of nontariff barriers to free flow of goods; (iii) Adopt rules of origin that foster trade and Outputs were due in 2012. No updates provided.
Achievement of Technical Assistance Outcomes 9 investment; and (iv) Develop simple, harmonized, and standardized trade and customs processes and procedures. Output 1. Improved understanding of the implications of membership of REI initiatives. (i) Updates on Timor-Leste s preparations for membership of ASEAN and other REI initiatives publicly disseminated by MOFA (at least two public releases annually in 2013 and 2014, 2012 baseline: 0); (ii) Submissions by MOFA on Timor-Leste s preparations for membership of ASEAN and other REI initiatives considered by the Council of Ministers (at least two submissions annually in 2013 and 2014, 2012 baseline: 0). Output 2. Enhanced policy framework for REI (i) Prioritized REI action plan and supporting timebound, results-based capacity development program shared with stakeholders (by mid-2013). (2012 baseline: No); (ii) At least half of the initiatives of the REI action plan implemented (by end of 2014). (2012 baseline: 0) Output 3. Improved communication with regional organizations and partner countries (i) Public dialogue events held in Timor-Leste are attended by regional organizations or ASEAN member state(s) (at least one event annually in 2013 and 2014, 2012 baseline: 0); (ii) Prioritized REI action plan submitted to the ASEAN Secretariat (by mid-2013, 2012 baseline: No); (iii) Practice established of informing the ASEAN Secretariat of improvements to Timor-Leste s ability to carry out the obligations of ASEAN membership (at least two submissions annually in 2013 and 2014, 2012 baseline: 0). No updates provided. No updates provided. No updates provided. ADB = Asian Development Bank; AIMS = aid information management system; ASEAN = Association of Southeast Asian Nations; ATP = Aid Transparency Portal; CPS = country partnership strategy; DDWSS = District Department for Water Supply and Sanitation; DGCS = Directorate General for Corporate Services; DMF = design and monitoring framework; DMTF = demand management task force; DNAF = National Directorate for Administration and Finance; DNAS/DNSAS = National Directorate for Water and Sanitation; DPMU = Development Partnership Management Unit; IEM = independent evaluation mission; INAP = National Institute of Public Administration; MOF = Ministry of Finance; MOFA = Ministry of Foreign Affairs; MOI = Ministry of Infrastructure; MoNRMEP = Ministry of Natural Resources, Minerals, and Energy Policy; MoPW = Ministry of Public Works; MoTC = Ministry of Transport and Communications; MPS = Major Projects Secretariat; NDS = National Directorate of Statistics; NDWSS = National Directorate for Water Supply and Sanitation; NDM = Directorate of Macroeconomy; NIPA = National Institute of Public Administration; NRW = non-revenue water; NUTL = National University of Timor Lorosae; OECD = Organisation for Economic Cooperation and Development; Q = quarter; REI = regional economic integration; SIP = sector investment program; SSTA = smallscale technical assistance; TA = technical assistance; TCR = technical assistance completion report; UNICEF = United Nations Children s Fund; WHO = World Health Organization; WS = water supply. Sources: ADB project documents, TA completion reports, ADB Independent Evaluation Department.