"Rehabilitation and sustainable development of fisheries and aquaculture affected by the tsunami in Aceh Province, Indonesia" (Project OSRO/INS/601/ARC) FINAL PROJECT EVALUATION 29 March 11 May 2010 Response draft dc 6 July 2010
A final evaluation of the project Rehabilitation and sustainable development of fisheries and aquaculture affected by the tsunami in Aceh Province, Indonesia (OSRO/INS/601/ARC) was carried out 29 March 11 May 2010 to assess project results and the quality of implementation. The evaluation methodology was based on five standard criteria (relevance; efficiency; effectiveness and impact; and sustainability) to be followed by the team during the evaluation s four steps: 1) desk study of project documents and other relevant literature; 2) individual or group interviews with project stakeholders in the nine districts covered by the project and Aceh s capital, Banda Aceh; 3) drafting of preliminary reports; and 4) briefings at the Ministry of Marine Affairs and Fisheries (MMAF) and FAO in its offices in Jakarta, Rome and Bangkok, with the double intention of hearing about project-relevant issues from stakeholders at these agencies and presenting preliminary conclusions from the evaluation to them. A draft report submitted dd June 2010 presented the evaluation team's findings, conclusions and recommendations, as well as comment on some wider lessons that had emerged from the project. This document is the Response to the evaluation, prepared in accordance with FAO guidelines Responsibilities and Procedures for Response and Follow-up Reporting on Evaluations - July 2008. The Final Evaluation Report is viewed by ARC, FAO and Indonesian government partners as reflecting well the activities, progress and challenges of the project. With only a few exceptions the recommendations are accepted. In the following table the key comments and recommendations made in the report are considered and addressed one-by-one. 1
ance by GENERAL 34. Increasing people s ability to use information technology (IT) received relatively little attention by this capacity building project. DKP and other institutions should be able to learn and implement various IT programs and processes to strengthen coordination and planning, including evaluation and monitoring. 37. Improved fisheries management and development urgently needs a reliable, accurate and representative data and information system at local level, not only on fish landings, but post-harvest issues as well. 39. There is strong potential for future replication of the project s approach of focusing on capacity building and collaboration in management planning and implementation, involving multiple agencies and organizations in communities affected by disasters or for any other specific initiatives aiming at empowering fish farmers to address poverty through This is true; IT training was not prioritized under the project as several other initiatives were already ongoing through UNDP and other international agencies and NGOs to improve IT skills of government staff. Some IT training was included under the project for the MMs, staff at the co-management centres, aquaculture livelihood centres and to the DKP data handlers. Parallel FAO projects also built IT skills of several government agencies linked to the fishing vessel registration programme DKP does have an existing data collection system that has value, despite its deficiencies in accuracy and coverage. In addition, some key fisheries advisers to the project take the view that good data does not necessarily lead to good management, and conversely, good management is possible with only limited data as long as they are the right kind of data. The project has demonstrated a successful approach to post disaster transition to development. Despite an initial wariness of stakeholders to a capacity building emphasis, by the end of the project, the approach had gained wide support. Government should include IT capacity building for staff in its work plans and budgets DKP should continue to strive to improve the accuracy and representativeness of its fisheries data, and MMAF should support improvements Agencies such as those of the UN and IFRC should consider replication of the project approach in other countries and post-disaster situations 2010-2015 MMAF, DKP, Badan Diklat 2010-2015 MMAF, DKP 2010-2020 UN, IFRC 2
ance by alternative income generation. Partnerships with multiple agencies are an efficient way of achieving shared goals. 42/169. Projects should emphasize integration of its components. Areas of activities such as coordination and planning and co-management should not be understood as separate technical specializations, but as common relevance for all components. 43/170. Project results, products and services such as those emanating from the survey on job descriptions and relevant recommendations made by the project should be taken up and used by DKP on district and province levels. The origin of the project was in an atmosphere of post-tsunami chaos where improved coordination and planning was recognized as being needed if sustainable and complementary work from a large number of agencies and NGOs was to be brought about. By project start-up this was no longer a critical issue and the emphasis changed to one of longer term improvement of government coordination and planning. The structure of the outcomes into four separate components meant that linkages between them took special effort, rather than occurring naturally. A different Log Frame design may have made closer integration between components more likely. A stronger Government role in managing the project would also have brought the technical component work under a stronger umbrella of planning and coordination. This is happening already. Further uptake and usage is to be encouraged. Government needs to continue to find more effective ways to be more closely involved in development projects funded by outside donors. In this project, individual government staff participated strongly and made key contributions to the success of the project, but the project was far from being jointly managed by FAO and DKP. DKP should continue to take up and use project results, products and services such as those emanating from the survey on job descriptions, post harvest, co-management and aquaculture 2010-2020 2010-2020 MMAF, DKP, Aceh Government DKP, Aceh province and district governments, Badan Diklat 3
ance by 44 & 171. Provincial DKP should provide incentives to districts to further enhance inter-agency coordination, and coordination among districts. DKP province also needs to give attention to coordination for SFM and comanagement, and aquaculture for the districts. 45 & 172. For success of the SMF in the coastal village, interventions should not focus solely on fishery livelihood activities. Other livelihood activities should also be supported, so that the fishery does not risk overexploitation as the main source of community livelihoods. At the same time overall community resiliency is promoted. Given the increasing autonomy of districts with delegated budgetary control, the task of DKP province is made more difficult in this area. Where the districts are motivated to work together because of common needs, such has been seen under the project with the issue of illegal fishing, coordination is more likely. DKP could budget joint meetings with districts however to continue to build links. Some attention has been paid to this under the project particularly in the case of the comanagement centres where women s self help groups were promoted and supported in two centres. This has enabled them to develop new livelihood options. Many NGOs have been active in the project target communities developing alternative livelihoods which made any project initiative less needed. Fisheries remains a key income provider however. The project also aimed to increase the income from fish landed by improving post-harvest handling and developing value added products. In aquaculture most farmers already have diverse sources of employment and income with the tambak providing only a portion of their total livelihood. DKP province (and district) should budget for soft issues such as coordination as well as for infrastructure improvements etc., and the parliament needs to be lobbied to approve such budget lines The Aceh provincial and district governments should support further work to diversify livelihoods in coastal communities 2010-2020 2010-2020 DKP, Aceh province and district governments. Aceh provincial and district governments, NGOs 4
ance by 46 & 173. DKP (both district and province) and the Agency for Food Security and Extension (BKPP) should make changes to budgetary policy with regard to maintaining extension services and provision of good infrastructure through tambak channel rehabilitation. DKP will have to increase its activities in the field in order to keep up the momentum created by the project. Such requests (digging/rehabilitation of water channels etc.) should be made on occasion of dissemination of project results to the province and district governments. DKP in Aceh Utara and Bireuen already have some plans to rehabilitate tambak canals in some project target areas, but the overall economics of rehabilitation work are questionable until the shrimp production from the tambaks becomes more reliable. The project has demonstrated that this reliability can be achieved by adoption of BMPs. The project extension work through the system of Field Facilitators and the farmer group structure now built through the Kontak Petambak and Petua Neuheun, working from the Aquaculture Livelihood Centres built by ADB ETESP, offers a successful model that the Extension Service may wish to examine further and adopt, adapting it to the constraints of their operating procedures. DKP should request budget for tambak canal rehabilitation firstly in the clusters that have adopted BMPs and demonstrated their ability to work together and improve production. The extension service should examine the results of the project extension approach and consider if it could be a more effective model for their work 2010-2015 DKP BKPP 47 & 174. To promote continuity and sustainability, as well as to make good use of competent and skilled personnel trained by the project, existing field facilitators should be encouraged to seek employment with private aquaculture input supply or trading companies, who, in turn, should proactively be encouraged to hire this personnel. This would enable them to continue providing technical services to fish farmers and may happen if and when those companies increase their investment, but this is unlikely to occur in the current investment climate in Aceh. Efforts have been made to encourage the ALSCs to consider financing the activities of the Field Facilitators and a few may yet find continuing employment in this way. Find a way to retain the technical support for farmers of the Field Facilitators 2010-2015 ALSCs, Petua Neuheun 5
ance by in Aceh. The field facilitators, which have been trained by the project, are good candidates to work with such companies. 48 & 175. The co-managing partners, that is, DKP and fisher and community organizations, should make renewed efforts to set up reliable, accurate and representative data and information systems at local level, which are urgently required for any good fisheries development planning. 49 & 176. Fish processing and value-addition faces two bottlenecks: Insufficient good quality raw material and insufficient business management capacities. While the former is more difficult to address, follow up action should be given to foster improved management of the fish processing businesses. This will also provide an important entry point for women in the fisheries sector. See 37. above See 37. above DKP already has plans to build on the project work with promotion of new value added projects, perhaps targeting one new product each year Promote improved management of post-harvest businesses 2010-2015 DKP 6
ance by 50 & 177. Dissemination of project results and experiences will be an important vehicle to maintain and expand the current momentum in promoting and replicating examples of improved fisheries management and development in Aceh. Apart from widely circulating the many materials produced by the project in all its priority areas of activity, this could include a special flyer on the project s role as a transition project between emergency resppnse, livelihoods rehabilitation and sector development. Funding for this from FAO Rome is under discussion Produce a special flyer on the project s role as a transition project between emergency response, livelihoods rehabilitation and sector development. 2010 FAO On logical frameworks 51 (&178). Project logframes should be better developed, more often reviewed and regularly updated. Particular and frequent attention should be given clearly differentiating between project objectives and project outputs; to identifying (and mentioning) target beneficiaries and main actors in the formulation of project goal and objectives; defining risks and elaborating proactively on risk management (instead of assuming possible risks as assumptions ); All these points are accepted as valid Improve Log Frame design in the ways recommended 2010-2020 FAO, ARC 7
ance by provision of baselines prior to setting of target levels and adaptation of targets throughout implementation. 154. Yet one evaluator found the LFA as not ideally formulated. In particular there still seems to exist certain confusion with regard to objectives, outcomes, and outputs. Also, it would have been beneficial to name the relevant actors in the formulation of project goal and objectives in order to clarify responsibilities. Furthermore, risks were insufficiently defined and rather formulated as assumptions ; however, this also meant that risk management or mitigation strategies were not proactively elaborated; interestingly this was done in the project exit strategy. And, finally, target levels for measuring project achievements were very cautiousand conservatively set, which complicated a real assessment of project success. On evaluations 52 (&179). Evaluation outlines and TOR should clearly differentiate between ongoing or final evaluations. Particular emphasis The End-of-Project Evaluation would have been improved if budget and time had allowed these recommendations to be followed. Budget provision for post-project impact Consider the recommendations in future project planning. 2010-2020 FAO, ARC, DKP 8
ance by should be put on joint implementation of the evaluation by the entire evaluation team. This should include start-up discussions on evaluation purpose and its utilization-focus, philosophy and methodology, as well as joint wrapup meetings to discuss, take note and agree on findings. For end-ofproject evaluations conduct a short study on project impact and sustainability after the project has ended. 95. Furthermore, the overall high delivery values suggest that indicators and target levels were chosen cautious- and conservatively, that is, within a very safe range. Many targets were exceeded, at times by very high margins. Also, while overall achievement is beyond 100%, there were large discrepancies between targets, some being excessively over-, others significantly underachieved. In future, targets should be more realistically set, and, if necessary, adapted during the course of project implementation. COMPONENT 1 evaluation is rarely made but should be. See 51 above See 51 above 9
ance by 9. While overall component efficiency was high with more than 100 percent output delivery, it was found wanting in as much as many activities were implemented with relatively limited involvement of the government. DKP faced constraints in fully participating as an equal partner in managing and implementing the project Government needs to continue to find more effective ways to be more closely involved in development projects funded by outside donors. In this project, individual government staff participated strongly and made key contributions to the success of the project, but the project was far from being jointly managed by FAO and DKP. 2010-2020 MMAF, DKP, Aceh Government 119. In general, the project implementation process was carried out efficiently and effectively, but support and follow up action from local government was inadequate. In the specific areas targeted, government support and follow up was quite good but he overall scope and impact was admittedly limited See item 9 above 114. While it was a good move to use the Aceh Green vision and strategy as a policy framework, it is a mechanism said largely linked to the present provincial government, with little impact on district levels. If the Governor is re-elected he may be able to extend his Aceh Green vision to the districts Promote Aceh Green to the districts 2010-2015 GAC, Aceh Govt. 115. There is no strong evidence from the field that the district governments are leaning towards conservation issues arising from the Aceh Green vision. See 114 above See 114 above COMPONENT 2 10
ance by 10. However, despite an overall output delivery of more than 90 percent, the evaluators considered the component only partly efficient, as many project activities were implemented with relatively limited involvement of particularly the provincial government 36. Sustainability of co-management requires time, resources and expertise. Considering the nature of cooperation among various stakeholders, including the government. For the immediate needs, it would be appropriate to be handled by NGOs, or related third parties. The most probable source of funds for this project should be coming from Provincial DKP and other sources 140. While at present community members show great enthusiasm to continue activities initiated by the project, conflicts over illegal fisheries and similar may impact their sustainability in the long run. COMPONENT 3 26. However, while some stakeholders assign this Component the highest impact, they question See 9 above See 9 above Support has already been generated at the Jakarta level with MMAF and two district governments have already made budget planning commitments to support the co-management centres Disagree. It is precisely over issues such as conflicts over illegal fishing that the comanagement groups have come together and proven most effective There is a future for extensive culture in Aceh, but incomes would remain low, They would improve with higher production levels. Support existing comanagement centres and promote new ones Support existing comanagement centres and promote new ones 2010-2015 2010-2015 Support expansion of BMPs 2010-2020 MMAF, DKP province and district, NGOs MMAF, DKP province and district, NGOs DKP, BBAP 11
ance by the long-term future of extensive aquaculture. 141. Additional inputs are required: finance, production inputs, seed banks don t provide such inputs COMPONENT 4 20. Unfortunately, improved postharvest and marketing has so far not been included as a focus in government policies. 27. Notable is the relatively low coverage when compared to the other components: From 64 operators and their households Given the history of the local industry however and the limited financial means of farmers, it is first necessary to make production more reliable through BMPs. Then farmers will be willing to make the necessary investments in their operations to raise production, and will have the technical skills to move to higher production methods. This has been the strategy of the agency consortium working with the tambak sector since the tsunami and it has proven effective through this first stage. Change of this kind needs years not months to effect and the industry will be ready to take the next step in 2011 provided the 2010 crop is successful. True, but project studies have shown that finance would be available from traditional lenders if the business was less risky. The problem is not so much availability of credit, but reliability of production. It is well embedded in national and MMAF policy, but it is true it has received little emphasis from the province or districts in Aceh. As a result of the project work though, budget to build on some of the project initiatives is to be included in future DKP plans, including for next year. There is good evidence that the value addition businesses started under the project will prove sustainable and expand. Major change in the post harvest sector will hinge on improved consumer awareness and this Build BMP programme 2010-2020 DKP, BBAP Greater focus needed from local government on improved post harvest and marketing Further promotion of project initiatives needed, such as new value added businesses, better handling, raising consumer awareness 2010-2015 DKP, Aceh Government 2010-2020 DKP, MMAF 12
ance by involved during the pilot phase to 89 in early 2010, or an increase of 40 percent operators who have taken up component results during the last three months. So far this may not be the critical mass necessary to enable medium- to long-term component sustainability. will take time 142. While the strong continuation of fish processing businesses despite withdrawal of project support to the pilot ventures in late 2009 is a good sign, this short period of selfsupported operations may not be sufficient to provide proof of medium- to long-term sustainability. Here sustainability of fish processing businesses may be impacted by lack of raw material and business knowhow. Possibly, but time will tell. Strengthen business environment in Aceh 2010-2015 Aceh Government MONITORING & EVALUATION 152. Unfortunately, no baseline KAP was carried out at the start of the Project, so no real measure of impact of the Project is possible. In our case, comparisons are made between targets and theoretically set targets and actually observed achievements. While the survey Agreed, the question remains. Circulate information about the KAP to other agencies and NGOs 2010 FAO, ARC? 13
ance by approach is interesting, the question remains if indeed the 2009 KAP survey results will ever be updated and used by another development intervention. 14