M&E on the cutting edge Improving the Use of M&E Processes and Findings Experiences from South Africa Mr Ismail Akhalwaya Programme Manager Institutional Performance Monitoring and Evaluation Department of Performance Monitoring and evaluation in the Presidency 20 March 2014 Wageningen, The Netherlands 1
1. COUNTRY BACKGROUND Pre 1994 2
1. COUNTRY BACKGROUND - Post 1994 Geograhic 3
1. COUNTRY BACKGROUND (continue) Semi-federal system 3 spheres national, provincial and local government - 9 provinces (states) with own governments but unitary state 278 municipalities Most development services provided by provinces (health, education, agriculture, social development) with local governments providing water, sanitation, electricity, planning (cities a bit wider) 4
1. COUNTRY BACKGROUND (continue) Total population= 52.98 million Population by race Africans= 42 284 100 (79.8%) Whites= 4 602 400 (8.7%) Coloureds= 4 766 200 (9%) Asians= 1 602 400 (2.5%) Government structure Federal Number of Ministries 47 Ministries 9 Provinces 5
Timeline 1994 1995 1996 1997 First democratic election Rugby World Cup TRC hearings National Anthem adopted National Flag adopted Land rights restitution Constitution adopted Robben Island Museum Presidential inauguration Motor Industry Development Programme (MIDP) Africa Cup of Nations 50th ANC conference Reconstruction and Development Programme (RDP) 1998 1999 2000 2001 TRC final report Elections Coat of arms adopted World Congress against Racism Voortrekkerhoogte renamed Thaba Tshwane isimagaliso Wetland Park declared World Heritage Site 13th international AIDS conference 7
2002 2003 2004 2005 Mark Shuttleworth first South African in space Towards a 10 year review Elections 50th Anniversary of the Freedom Charter World Summit on Sustainable Development Hector Pieterson Museum opens Cricket World Cup Announcement of SA hosting 2010 World Cup TRC report released Expanded Public Works Programme launched Charlize Theron wins Oscar Gold for 2004 Olympic Games swimmers Southern African Large Telescope (Salt) in Sutherland inaugurated 2006 2007 2008 2009 Same-sex unions SA serves on United Towards a Fifteen First Mandela Day legalised Nations Security Council Year Review Tsotsi wins Oscar Freedom Park opens Natalie du Toit Elections qualifies for Beijing Olympics Rugby World Cup 8
2010 2011 2012 2013 FIFA World Cup SA joins BRICS Gautrain starts running Framework of the New Economic Growth Path 50th anniversary of Human Rights Day Third Census since democracy 17th Conference of the Parties (COP17) SKA bid winners announced African Union Commission chairperson Nelson Mandela bank notes released National Development Plan launched Announcement of Infrastructure plan Team SA wins 6 medals at Olympics Smart ID cards 1913 Natives' Land Act Centenary Nelson Mandela remembered Mandela movie 'Long Walk To Freedom' Development Indicators 2012 released 2014 20 Years Review 9
MAIN FEATURES OF GOVERNMENT PERFORMANCE MONITORING
2. WHAT IS THE PROBLEM THAT M&E AIMS TO ADDRESS? Need for continuous improvement, but.. Poor planning, weaknesses in setting of indicators and targets, weak logic models / theories of change Focus on activities not outcomes, without assessing the results or impact of the activities Monitoring and reporting for compliance Sporadic evaluation so no in-depth learning Insufficient use of data to inform improvements Evidence-based planning and decisionmaking not sufficiently valued Lack of reengineering of plans and business processes based on evidence Implementation weaknesses Disappointing results - lack of progress in key indicators, such as rural unemployment Poor quality of service delivery Insufficient value for money Poor accountability 11
3. WHY M&E? Improving policy or programme performance (for continuous improvement): this aims to provide feedback to programme managers. Improving accountability: where is public spending going? Is this spending making a difference? Improving decision-making: Should changes be made to the intervention? Should it be continued? Should increased budget be allocated? Generating knowledge (for learning): increasing knowledge about what works and what does not with regards to a public policy, programme, function or organization. 12
4. CULTURE -BASED BARRIERS Problems not treated as an opportunity for learning and improvement Senior management do not champion M&E and honesty about performance M&E is regarded as the job of the M&E unit, not all managers There is not a strong culture of M&E in the department M&E is seen as policing and a way of controlling staff The M&E unit has little influence in the department 45% 44% 40% 39% 39% 54% Fear of admitting mistakes or problems 33% The hierarchy makes it difficult to openly and robustly discuss performance Little respect for evidence-based decisionmaking in the department Resistance from officials to transparent decision-making processes 13% 27% 29% Problems are concealed 13% 0% 10% 20% 30% 40% 50% 60%
5. ROLES AND RESPONSIBILITIES FOR PLANNING AND M&E IN SA Auditor General Independent monitoring of compliance Auditing of performance information Reporting to Parliament Public Service Commission Independent monitoring and evaluation of public service Focus on adherence to public service principles in Constitution Reporting to Parliament Constitutional power Legal power Executive power National Treasury Regulate departmental 5 year and annual plans and reporting Receive quarterly performance information Expenditure reviews Cooperative Governance Dept (DCOG) Regulate local government planning Monitor performance of local government Intervention powers over local government Public Service Dept (DPSA) Monitor national and provincial public service Regulate service delivery improvement Presidency National Planning Commission (NPC): o Produce long-term plan (20 years) Department of Performance Monitoring and Evaluation (DPME) o Produce governmentwide M&E frameworks o Facilitate government 5 year plans for priorities o Monitor and evaluate plans for priorities o Monitor management practices of government o Monitor front-line services Line depts (national/prov) Monitor sectors 14
6. TIMELINES 2010 DPME established. 12 outcomes agreed, Minister s performance agreements, delivery agreements for each outcome, quarterly reports start, using traffic light system 2011 Systems for Management Performance Assessment (MPAT) created, monitoring of front-line services, and Evaluation Policy Framework. MPAT assessment of 103/156 national and provincial departments 2012 Reports against all of these programmes started including the 1 st evaluation completed. MPAT assessment of all 156 national and provincial departments 2013-14 Systems for monitoring local government, citizen-based monitoring and operations management being piloted. Currently consulting on an Discussion Document on Performance M&E Principles and Approach 15
7. FOCUS OF DEPT OF PERFORMANCE MONITORING AND EVALUATION (DPME) M&E of national priorities (Outcomes) National evaluation system Management performance assessment M&E of front-line service delivery Unannounced visits Presidential Hotline Citizen based monitoring Monitoring of local government 16
The Outcome System Government has adopted 12 outcomes with a view to measurable performance and accountable delivery. At the end of April 2010, the President signed performance agreements with all Cabinet Ministers. In these performance agreements, Ministers were requested to establish and participate in Implementation Forums for each of the twelve outcomes. The Implementation Forums have developed delivery agreements for the Outcomes. All departments, agencies and spheres of government that are involved in the direct delivery required to achieve an outcome, are party to the agreement. The PoA system monitors progress of the delivery agreements. It tracks and reports on the key aspects through indicators and targets for the outputs, suboutputs and in some cases, activities). A dedicated website for the PoA 17
18
National Evaluation System While monitoring tracks what you have planned to do, evaluation asks deeper questions of effectiveness, efficiency, causality, relevance and sustainability. Government is taking forward evaluation to improve government s performance and development impact, accountability, decision-making and to widen the knowledge base around government s work. A National Evaluation System is being established to entrench the quality and the use of evaluations in government, led by the Evaluation and Research Unit in DPME, and supported by a cross-government Evaluation Technical Working Group. The first National Evaluation Plan 2012/13 has the strategic priorities for evaluation for 2012/13 and was approved by Cabinet on 13 June 2012. The second National Evaluation Plan for 2013/14 to 2015/16 was approved in November 2012 and the evaluations are being started. The 2014/15 evaluations have been selected, and approved by Cabinet 19
20
Management Performance Assessments (MPAT) DPME has been mandated to regularly assess the quality of generic management practices in departments. A methodology for doing this was developed, in collaboration with the Offices of the Premier, National Treasury and the DPSA, and in consultation with the Office of the Auditor General and the Office of the Public Service Commission. The methodology has been informed by similar management performance assessments carried out in other countries. The methodology involves working with the management of national and provincial departments to carry out self-assessments which are then moderated by subject matter experts and by cross-referencing to data produced by bodies such as the Auditor General, Public Service Commission, National Treasury and DPSA. 21
Levels of assessment Level Level 1 Level 2 Level 3 Level 4 Description Non-compliance with legal/regulatory requirements Partial compliance with legal/regulatory requirements Full compliance with legal/regulatory requirements Full compliance and doing things smartly 22
23
Frontline Service Delivery Monitoring Programme of unannounced visits to service sites such as schools, health facilities, social grant facilities, police stations, and municipal customer walk-in centres. The objectives are to collect evidence on the quality of services and to work with the relevant departments to demonstrate how to use such monitoring information to inform improvements in the quality of service delivery 24
User Staff Monitor User Staff Monitor User Staff Monitor User Staff Monitor User Staff Monitor User Staff Monitor User Staff Monitor User Staff Monitor National overview of monitoring visits (April-December 2013) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Location & Accessibility Visibility & Signage Queue Management & Waiting Times Dignified Treatment Cleanliness & Comfort Safety Opening & Closing Times Complaint Management System 25
26
Presidential Hotline Set up in 2009 to allow citizens to log their complaints and queries regarding service delivery. To date, more than 180 000 cases have been logged and assigned to the relevant departments and agencies for resolution, with 94% of cases having been resolved On a monthly basis, departments are informed of their responsiveness to complaints in the form of a scorecard on progress in case resolution and they are informed of the outcome of satisfaction surveys conducted of the quality of their complaints resolution 27
Citizen-Based Monitoring (CBM) CBM describes an approach to monitoring government performance that focuses on the experiences of ordinary citizens in order to strengthen public accountability and drive service delivery improvements. After the approval by Cabinet in August 2012 of a Framework for strengthening citizengovernment partnerships for monitoring of frontline service delivery 28
8. LESSONS OF EXPERIENCE The importance of consistent communication with users, continuous training on the use of models, improves relations and buy in for the use of these models (Outcomes and MPAT) Documenting experience that emerge from the application of tools, enhances shared learning and ease of improvement / adjustment of tools Creating a culture of partnership and support, builds ownership and interest in the use of the tools More discipline and commitment in the application of legislation and prescripts, and improved knowledge of users of these prescripts (MPAT) Improved understanding of the need to develop more impact indicators and improve the lives of the citizens 29
THANK YOU 30