Early experiences with the Equitable Impact Sensitive Tool - EQUIST

Transcription

1 Early experiences with the Equitable Impact Sensitive Tool - EQUIST December 2017 Health System Strengthening Unit Working Paper UNICEF Health Section, Programme Division Unite for children

2 EQUIST Formative Assessment: Experience from 35 countries United Nations Children s Fund (UNICEF), New York Health System Strengthening Unit, Health Section, Programme Division UNICEF 3 UN Plaza, New York, NY January 2018 This is a working document. It has been prepared to facilitate the exchange of knowledge and to stimulate discussion. The findings, interpretations and conclusions expressed in this paper are those of the authors and do not necessarily reflect the policies or views of UNICEF or of the United Nations. The text has not been edited to official publication standards, and UNICEF accepts no responsibility for errors. The designations in this publication do not imply an opinion on legal status of any country or territory, or of its authorities, or the delimitation of frontiers. i

3 Health System Strengthening Unit Working Paper UNICEF Health Section, Programme Division Early experiences with the Equitable Impact Sensitive Tool Experience from 35 countries Diana Sera, UNICEF HQ Ben Nemser, UNICEF HQ Comments may be addressed by to the authors: Diana Sera cc: David Hipgrave and Lakshmi Narasimhan Balaji ii

4 Acknowledgements The authors would like to thank everyone who participated in this study by sharing their valuable insights for this formative assessment of EQUIST, including partners from UNICEF, UNFPA, USAID, WHO, NGOs, Ministries of Health and academia. We are also especially grateful to our donors, especially at the Bill and Melinda Gates Foundation (BMGF) for their generous financial support to UNICEF for the scale up of this initiative. UNICEF s Health System Strengthening Unit (David Hipgrave, Claudia Vivas, Lakshmi Narasimhan Balaji and Shahrouh Sharif) assisted with the design of this assessment and reviewed and commented on drafts of this paper. Keywords: health systems strengthening, EQUIST, formative, assessment, equity iii

5 Contents Acronyms... v Executive Summary... vi 1. Introduction Background Purpose of Assessment Key questions included in the Assessment Methodology Data Collection Sampling and Triangulation Study limitations Findings Demographics Summary of main findings by assessment criteria Conclusions Recommendations Annexes Country applications Theory of Change Evaluation Matrix Interview and Discussion Guide Online survey tool Respondent classification iv

6 Acronyms ANC ARI CHAI DHS EQUIST HSS HMIS LiST RiHIS M&E MBB MNCH MICS MODA MoH NGO ToR UNICEF UNFPA WHO Antenatal care Acute respiratory infection Clinton Health Access Initiative Demographic and Health Surveys Equitable Impact Sensitive Tool Health System Strengthening Health Management Information System Lives Saved Tool Routine Health Information System Monitoring and Evaluation Marginal Budgeting for Bottlenecks tool Maternal, Neonatal and Child Health Multiple Indicator Cluster Survey Multiple Deprivation Analysis Ministry of Health Non-Governmental Organization Terms of Reference United Nations Children s Fund United Nations Population Fund World Health Organization v

7 Executive Summary Through interviews and document review, an assessment of the use of EQUIST for strategic planning and prioritization of maternal, newborn and child health (MNCH) interventions and allocation of related resources was conducted between August-November The assessment explored the extent to which EQUIST and related policy dialogue had served to institutionalize, among decisionmakers in low- and middle-income countries, an equity focus in the planning, prioritization and allocation of resources for MNCH interventions. A total of 124 individuals in 35 countries, from UNICEF, UNFPA, USAID, WHO, NGOs, Ministries of Health and academia, responded to the assessment; the majority (99 or 80%) from 30 countries had participated in an EQUIST training. The analysis drew on all individuals that responded to the survey including those that had received any form of training or orientation on EQUIST. Overall, given that EQUIST had just been rolled out in most of the countries, respondents felt it was too soon to measure its impacts. Nonetheless, the assessment was able to establish the following: 1) Relevance: The tool was said to improve and strengthen decision makers focus on the most marginalized. In particular, a large majority indicated that EQUIST had positively influenced country policies, processes and procedures in health sector planning and priority-setting. 2) Effectiveness: The respondents were motivated towards the equity-based approach to planning and the use of the EQUIST tool. EQUIST was also noted as instrumental in informing the development of national plans, UNICEF Country Planning Documents (CPDs) and investment cases particularly for RMNCAH. However, the tool was seen to be less helpful in informing resource allocation compared to other tools such as the One Health. Among the key factors and conditions associated with the effective use of EQUIST for planning were the quality of EQUIST training, data availability and quality, follow-up/mentorship support and the capacity of the local health system. 3) Timeliness: Overall, support provided was timely, however, there were several suggestions to improve on-time delivery of EQUIST country support. 4) Sustainability: Findings indicate that EQUIST will be unsustainable without ongoing UNICEF support. Furthermore, 70% of respondents agreed that there was limited systematic documentation and dissemination of good and bad practices or lessons learned on EQUIST. Moreover, most respondents did not identify successful experiences where EQUIST had been linked to the mobilization of additional resources (although the recent roll-out of training may impact this result). Lessons learnt and suggestions to improve use of EQUIST include: 1) Strengthening efforts to advocate for and market the tool both at country and at global levels. 2) Improving the quality of the EQUIST training e.g. by increasing the duration of the training, providing mentorship support, follow up trainings and including the use of local trainers, applying the use of continuous online learning using webinars, step by step videos and updates from the EQUIST core team, and the sharing the training documentation ahead of time. 3) Improving the availability and quality of EQUIST data. 4) Enhancing the complexity of EQUIST modeling by, for example: including multiple deprivation features so that different indicators can be overlapped to show different dimensions of deprivation of outcomes. 5) Adding adolescent health and nutrition indicators and other thematic focus areas to the tool. 6) Systematically documenting good and bad practices on the use of EQUIST vi

8 7) Linking the tool to implementation through conducting research informed by the tool 8) Improvement in EQUIST s costing abilities or aligning it with other costing tools, such as OneHealth. To a degree, objectives around building capacity and developing equity-based strategies / plans have been met; however, EQUIST training needs enhancement and the tool needs to be sustained and lessons well documented. The drive and commitment of UNICEF s HSS team, funding from the Bill and Melinda Gates Foundation (BMGF), and the motivation and interest of the users for the approach have contributed to these results. vii

9 1. Introduction 1.1 Background In the aftermath of the Washington Child Survival Call to Action in June 2012, many countries pledged to review their national policies and plans to identify and prioritize strategies to accelerate progress towards the ambitious goal of below 20 young child deaths per 1,000 live births by To facilitate country processes in support of governments commitment, the BMGF funded UNICEF to develop the Equitable Impact Sensitive (EQUIST) web-based tool. The tool intended to establish a simple and user-friendly interface. It incorporates functionality of Lives Saved Tool (LiST) in its backend to model lives saved and uses data and principles from MBB for its costing component. EQUIST identifies cost-effective interventions, prioritizes key bottlenecks that constrain their coverage and targets the most effective and equity-focused strategies that can address these bottlenecks to increase MNCH intervention coverage and fast track the reduction of maternal and child mortality, especially among deprived populations. The main global objectives of the BMGF support are to: (a) develop a web-based EQUIST software platform; and (b) build the capacity of global and regional facilitation teams and supporting partner institutions (e.g. regional academic institutions, etc.); and (c) develop advocacy and support materials and approaches to facilitate the dissemination and use of EQUIST for enhancing policies, programs and investments. At country level, UNICEF has been working with governments to adapt and apply EQUIST as an evidence-based and equity-focused planning tool to improve country strategies and plans in order to raise the coverage of high impact maternal, newborn and child health (MNCH) and nutrition interventions. The application of EQUIST involves: Building Capacity: developing the capacity of technical and managerial personnel within UNICEF Country Offices, government officials and key national partners; Conducting Scenario Analyses: supporting stakeholders in identifying priority populations, interventions, bottlenecks and operational strategies, and estimating the required additional resources; Facilitating Dialogue: facilitating policy dialogue among key decision makers, to support key decisions on priority interventions and strategies; and Documenting Lessons Learned: capitalizing on the lessons learned in the process through: 1) systematic documentation; and 2) peer-to-peer experience sharing (e.g. community of practice, learning tours, coaching and mentorship). As a result of these activities, countries have the opportunity to: o Develop evidence-based, equity-focused strategies and enhanced plans for MNCH and nutrition; o o Mobilize and re-allocate resources to address remaining gaps in strategic plans; and Institutionalize an equity-based approach to planning and health systems strengthening (HSS). Based on the feedback received from its first phase of use at country level, a new version of EQUIST is being developed. A contract for related work was signed with Community Systems Foundation (CSF) in September The new version of the tool will have numerous enhancements, including the possibility to include adolescent health, use district level data, exchange data with other tools 1

10 and platforms, and allow visualization of georeferenced information. EQUIST 2.0 will be launched in early The HSS team has updated and expanded the information on national and sub-national health status included in the EQUIST national databases; information from Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) has been included for over 60 countries. Additional information from facility surveys has also been incorporated for a subset of countries. A number of high-profile EQUIST dissemination activities have been implemented. There was a session fully dedicated to EQUIST at the Fourth Global Symposium on Health Systems Research in Vancouver in November 2016; a booth was set up at the Symposium to demonstrate the use of the tool. Over 100 researchers, government officials and international development professionals participated in the session, and many more visited the booth. Furthermore, there was a dedicated half-day session on EQUIST at the Institutionalizing Community Health Conference, held in Johannesburg, South Africa in March In addition to presenting the tool, country delegations from Senegal and Malawi provided testimonials highlighting their experiences using EQUIST as a means to better understand and demonstrate the role that community health strategies can play in addressing health system bottlenecks for deprived populations. The pilot phase of country applications has continued, with support provided to Senegal, Malawi, Bangladesh, Sudan, Mozambique, Cameroon, Uganda and Djibouti, Panama, Guinea and Guinea Bissau in the context of the Global Finance Facility (GFF) and other activities. In addition, a growing number of countries are requesting support including, Egypt, Morocco, Myanmar, Rwanda, Guinea- Bissau and Guinea-Conakry among others. Recent application of EQUIST in Bangladesh was undertaken at division (sub-national) level, enabling an equity analysis at district-level. This is relevant for district planning, and comprised the first use of EQUIST with administrative data collected using the standardized administrative data-collection tool, District Health Information Software 2 (DHIS2). In short, the number of country experiences with EQUIST is growing, and this merited an assessment of the related processes, the influence of the tool, and preliminary results. 1.2 Purpose of Assessment The primary purpose was to conduct a process evaluation of the use of EQUIST for strategic planning and prioritization of MNCH interventions and allocation of related resources. UNICEF sought to learn by documenting the current use of EQUIST through: (i) a compilation of experiences on inputs and processes undertaken to date, and (ii) to the extent possible, identifying outputs and emergent outcomes. This method of assessment involved documenting the experience with EQUIST to date, through independent compilation of experiences in selected countries where it had been applied, and of individuals who had received the training. The two purposes listed were met by examining whether, and the extent to which, EQUIST and related policy dialogue had served to institutionalize among decision-makers in low and middle-income countries an equity-focus in their planning, the use of evidence in prioritizing decisions on resource allocation, and allocation of resources for MNCH interventions. 1.3 Key questions included in the Assessment This assessment entailed examining a number of questions related to facilitated equity-focused policy dialogue and the use of the EQUIST tool, including: 2

11 1) Retrospectively, have the country-level objectives been met (e.g. building capacity, developing equity-based strategies / plans; mobilizing resources for MNCH; and institutionalizing data-driven, equity-based approaches) and what has been achieved so far? 2) What factors contributed to success or failure of the policy dialogue and use of EQUIST what worked or had not worked. 2. Methodology 2.1 Data Collection The assessment team primarily used qualitative methods for data collection and analysis and triangulated the information drawn from each method. The data sources consisted of; desk review of documentation, key information interviews and an electronic survey. In addition, country case studies on the application of EQUIST were solicited and prepared from countries (see annex 6.1 for the country case studies). The assessment sought and gathered evidence available to test and validate the theory of change (see annex 6.2), and to answer the questions listed in the evaluation matrix (see annex 6.3). The key data collection methods used are elaborated in the text below; Desk Review: A comprehensive list of relevant materials together with electronic copies of key documents were shared with the team. These materials included proposals, trip reports, EQUIST guidance, pilot country reports and presentations, and other materials generated from the application of the tool. Financial data and health management information systems were not assessed. Interviews with key informants: Interviews were conducted with UN staff; and country decision-makers, programme managers who have been engaged in EQUIST piloting to date. Country field visits were conducted to two countries, Senegal and Uganda. The data collection tool (see annex 6.4) was refined during the pilot visit to Malawi. Electronic survey: This assessment principally drew on the results from an online survey of 88 respondents from 35 countries, particularly those where a field mission was not possible. These countries included: Haiti, Bolivia, DRC, Guinea, Guinea-Bissau, Bangladesh, Sudan, Djibouti and Uganda. The survey tool is presented in annex 6.5 and the full list of countries is in annex 6.5. Table 1: Respondents and Data Collection Methods used Data Collection Method # of Respondents 1. Key informant interviews Online Survey Training feedback forms 20 Total 124 3

12 2.2 Sampling and Triangulation This assessment did not make use of a randomized, statistically valid sampling process. Instead, a purposive sample was used. A field visit was conducted in Senegal and Uganda, with key informant interviews purposively identified through consultation between the evaluation team and the coordinating agency in country to ensure a diverse group of key informants. The assessment relied on triangulation both across and within categories of data sources. For example, the evaluation triangulated responses of different key informants at country and global level to ensure that differences were reflected in the analysis. The focus of triangulation was the testing of the causal assumptions relating to each of the evaluation questions. 2.3 Study limitations Given the recent training and roll-out of EQUIST, it was difficult to demonstrate impacts or long-term improvements. Furthermore, the available data varied from country to country as the county response was quite different. EQUIST use across countries varied widely and outcomes depended on priorities identified and implemented. Care was taken in the assessment of EQUIST contributions overall, given the plethora of other data initiatives. Selection bias may have occurred because we are using a purposive sample of key informants who may also have had a recall or motivation bias. On-line surveys often have low response rates, and therefore may not contribute significantly to the analysis. The assessment team sent reminders to boost the response rate which resulted in responses from 88 respondents (a response rate of 10%). 3. Findings 3.1 Demographics We enumerated 124 individuals, close to 60% of whom were male. These included: health specialists and chiefs, lecturers, Researchers, monitoring and evaluation officers and Directors of Health Systems, Planning and Policy and nutrition specialists. Close to half (49%) were from the UN system while the rest from government, civil society, and academia. Table 2 indicates the breakdown of respondents by organization type. Of the 48 UNICEF respondents, 41 (85%) were from country offices. Respondent details are elaborated in a table in annex 6.5 Table 2: Respondents by Organization Type Organization type # Percentage UNICEF 48 39% Academia 24 19% NGO 22 18% GOVT 17 14% Other UN Agency 13 10% Total % 4

13 3.2 Summary of main findings by assessment criteria EQUIST Training The analysis and results in this report, draw from the responses of individuals that had received some form of training/orientation on EQUIST. Altogether 78% of the respondents surveyed (99 of the 124) had been trained on the use of EQUIST. The training averaged 2-3 days and ranged from two hours (e.g. for an orientation) to two weeks. The trainings took place between 2015 and 2017, the majority having been during The trainings took place in several countries including: Lebanon, Uganda, Djibouti, Uganda, Sudan, Nigeria, Australia, and Senegal. Several participants indicated that they had been trained as part of the UNICEF HSS training in Melbourne. The individuals trained were from several organizations including; USAID, UNICEF, World Bank, CHAI, UNFPA and others. The number of people per training varied by country. The US had the largest cohort of trainees followed by Nigeria. UNICEF also made the largest organizational cohort of the trainees. In general, as indicated in figure 1, the users had favorable perceptions of the training and all claimed to be motivated and better off after the training. Respondents credited the facilitators as being experienced and more than half indicated that the EQUIST training contributed to capacity building for planning and priority-setting. However, the other respondents had just been trained and were still in the preliminary stages of the use of the tool. Figure 1: EQUIST Training Responses 5

14 Here below are some anecdotes of the EQUIST training experiences: Because of the training, I can conduct a situation analysis to identify health problems and marginalized populations to guide intervention design and strategy. I can also conduct a scenario analysis and scenario comparisons and assess impact of proposed interventions to help guide resource allocation decisions - Lecturer at an academic institution We first model interventions before implementation and drop interventions if impact is likely to be less - Researcher at an academic institution We use the tool to categorize country states according to the under-five mortality indicator and other indicators, identify the key bottlenecks behind this weakness and their root causes and plan to address those root causes in an equitable way. - Director of Planning and Policy at a Ministry of Health It has contributed in improving my capacity in the concepts of causality analysis and equity based approach in priority setting and planning but the actual dependence on the tool to set the priorities will depend on our efforts to update the data for Sudan to help in this process. - - Director of Planning and Policy at a Ministry of Health With regards to the quality of the EQUIST training, the majority of respondents (96%) interviewed reported that the training was adequate. Some of the reservations indicated include; the training duration was insufficient, the training did not cater for our needs, we were trained too early (6 months prior to going live with the system, ) and we were not provided any mentorship or follow up after the training. Only 32% agreed that they were able to use the tool on their own for in country for planning and priority setting, while the large majority (63%) indicated that they needed external TA to use the tool. There were favorable perceptions on the adequacy of the distributed training materials, with 89% with the view that the materials were adequate. For example, some viewed them as helpful as instructional materials for teaching in academia. Some individuals noted the need to simplify the training manuals and update the materials with context specific examples and scenarios. Furthermore, the EQUIST training procedures were seen to be complex for mid-level computer users. As a result, for the majority, follow-up trainings or follow-up from HQ on support on the use of the tool was suggested. Furthermore, some few individuals indicated they did not receive any training materials. When I was a trainee, there was nothing except the detailed technical manual, which is largely out of date now and contained many inaccuracies according to CSF. The training video prepared for the Vancouver HSG conference was very helpful and I have used that in my subsequent training. - Head of Unit & Senior Lecturer at an academic institution Another proposal is the need for a more elaborate training session on costing. The existing training materials and user guides also need to be updated - particularly for the costing/cost effectiveness module. A proposal was also made for the EQUIST training team to plan to hold different levels of training for example; basic, intermediate and advanced levels on the use of EQUIST, while taking into account proficiencies on LiST and OneHealth Tool. 6

15 The technical notes also need to reflect the changes that have occurred, and if possible, the notes should be circulated to the training recipients ahead of the training. The issue of scenarios was said to be somewhat difficult to understand. With regards to the identification and development of strategies and interventions in the tool, some of the options indicated in the tool, were noted as seemingly stereotyped. It was thus suggested to leave room for different options that may not be included in the tool. The approach to strategies to overcome bottlenecks in the EQUIST tool, was said to be too simplistic for meaningful use or dialogue. Thus, it was also suggested to provide better guidance/more information on the approach s pros and cons of the bottleneck analysis and the factors to determine the choice in different scenarios- Determinant Approach, Tanahashi or the combined one. Furthermore, the EQUIST database for some countries was as old as 2014, which was not realistic for planning and priority setting. It was also suggested to have more than one trainer per workshop, and to provide or further develop the capacity of multiple trainers, so they can handle other sessions instead of having one person conducting the entire training throughout the workshop. Relevance Fig 2: Relevance of EQUIST Figure 2: Relevance of EQUIST As in figure 2, there was consensus, among 99% of respondents that had been trained, that EQUIST had been had positively country policies, processes and procedures in health sector planning and priority-setting. Some agreed that the tool was user friendly, pragmatic, cost saving, scientific and revolutionary- beyond the traditional approaches used. Overall, (97%) of the respondents trained agreed that EQUIST had improved and/or strengthened decision-makers focus on the most marginalized by providing evidence based data on where to prioritize. Furthermore, there was agreement that equity-enhancing approaches saved more lives, 7

16 and were more cost effective. Investing in the poorest is both a moral obligation and a sound financial investment. EQUIST was said to allow health sector planners to go through an evidence-based planning process at both national and sub national levels. This includes: the identification and prioritization of key health problems, marginalized populations and causes of bottlenecks to overcome in the implementation process, leading to the scenario analysis and scenario comparison features of the tool. EQUIST was credited for allowing planners to go through a comprehensive process which ideally would lead to strategic planning and resource allocation to the most deprived, and hence equitable and rational allocation of resources. Ultimately, the tool is useful in presenting the impact of the intervention or strategy. Moreover, it is relevant in setting targets and estimating the likely impact of a proposed intervention from the best available evidence. The tool can also be updated with further relevant evidence based in order to attain the most reliable impact of interventions and bottleneck reduction strategies. EQUIST was credited by 74% of respondents as timely and in line with ongoing country planning processes. For example, it was seen to be helpful to RMNCAH Investment Case preparation in several countries including Mozambique and Madagascar. In Kenya, the sub-national level trainings aimed to inform the development of investment cases for use on advocacy with the county governments for better allocation to the health sector. Moreover, the tool had been helpful in the development of UNICEF s Country Planning Document (CPD) in several countries. In Nigeria, the tool was helpful in informing a situation analysis of maternal and under-five mortality in the six zones of Nigeria. The outcome was successful, and a paper was submitted for journal publication. Those familiar with LiST appreciated how EQUIST drew on LiST for its impact modelling. Here below are some additional testimonies of EQUIST: Very relevant. Especially in Kenya where planning and allocation of resources don't often follow the evidence of the magnitude of a problem, regional inequities and possible returns on investment/impact modelling scenarios. - Health Officer at a development and aid agency It s always very critical to inclusively target the marginalized population as part of the general planning process rather than incurring additional costs to plan for them at a later stage (often times when the interventions or strategies have shown no impact). - Research Fellow at a Centre of -Excellence for Maternal, newborn and Child health, at an academic institution In the reality the poor and the hard to reach populations are forgotten and the EQUIST tool will enable to put the focus on those populations and to have a real impact in the planning. - - CSD, Health Specialist-Health Officer at a development and aid agency I am very convinced that planning for health should be based on equity measures and that EQUIST can help a lot in this if the countries update their data annually to depend on it for this purpose. - Director of Planning and Policy at a Ministry of Health The feedback from the Health Management Teams is that they started looking at their data differently and were grateful for the emphasis of equity. There are huge disparities that exist even within Counties and it was useful that with the training, the health managers started 8

17 thinking about who is being left behind and return for investment and prioritization for interventions. - Monitoring Officer at a development and aid agency Some argued that the tool was best used for situation analysis, however, not for costing or budgeting. Other tools such as the MBB and One Health tool for costing were credited for the costing use. It is important to understand that there are other tools that could be used to supplement. A driver with one tool is not a very well equipped driver to be on the road. The other tools noted are: the Health Equity Analysis Tool (HEAT), One Health tool, DHIS 2 and LiST. - CSD Specialist at a development and aid agency Another crucial point revealed from this assessment is the need to broaden the EQUIST s area of focus as the health sector contributes to only 20% of the population health; rest 80% are determined by socio-economic factors, health behavior and physical environment1. Another emerging area is self-care2. Disaster risk reduction and climate change are other areas. These areas, if integrated with EQUIST, will make the tool stronger. Clarity was sought on the costing assumptions as well as the assumptions made leading up to the estimation of the effectiveness of the various strategies to reduce bottlenecks. Moreover, issues noted including; the lack of data or inaccurate data in the tool, may necessitate the review of the tool s assumptions. Furthermore, the automation of all solutions (pre-determined options available for selection) may not allow specific context information to be addressed. Furthermore, one respondent suggested that the purpose of EQUIST should go beyond highlighting extant problems with resource allocation as highlighted below; EQUIST mainly lends itself to the thinking about the enhanced impact of such allocations but does not guide the planning for restructuring allocations in national budgets or subnational use of the resources, and hence misses a practical step which is necessary for the equity-based approach to lead to change. The expectation that better RMNCAH Investment Cases will lead to change is fine when linked with a GFF loan/grant opportunity; however, the use for restructuring existing government funding or priorities is more difficult - Senior Economist at an International NGO 1 Magnan S, Fisher E, Kindig D, et al; Institute for Clinical Systems Improvement. Achieving Accountability for Health and Health Care. July

18 Implementation Effectiveness As indicated in Figure 3, only 27% had used EQUIST or a related equity-based approach to develop national and subnational investment cases. For most individuals, this work is still in progress. For most countries, EQUIST is meant to inform the health strategic plans and the annual work plans for the next 5 years. Only 51% were aware of changes made to national, district or partner plans as a result of the use of EQUIST or an equity-based approach to planning. The majority (74%) agreed that EQUIST had helped to provide timely inputs to policy and programme decision-making. EQUIST tool was used mostly at national levels to build the case for addressing inequities and generate costing scenarios for comparison, therefore helping governments to better allocate the resources. However, it may not yet be useful at the sub-national levels due to lack of sub-national data at this level for many countries. The costing needs to be updated/more reasonable as it seemed to be very high per capita. It was also proposed to display the costing prior to running through LiST so that the scenario analysis could be less trial and error with respect to budgeting. Regarding, UNICEF s ability to provide adequate and timely support for the use of EQUIST or an equity-based planning approach, 52% agreed that this was the case. Timeliness (i.e. on-time delivery) needs to be improved. Respondents suggested the need for further follow up on the trainings that had been conducted, as illustrated by the quote below: A 2-day training with no follow-up may not give much results, especially in a country like Haiti where you have to give a very close support otherwise colleagues forget really quickly what they've learned. - Staff at the at a development and aid agency Fig 3: Implementation Effectiveness of EQUIST 10

19 Sustainability With regards to sustainability, about 46% of the respondents agreed that sustainability considerations (technical, financial, institutional) had been integrated into implementation of the equity-based planning approach and use of EQUIST, while another 36% differed on this. In some instances, sustainability considerations were said to be part of ongoing discussions at country level, especially with the ongoing development of the next country program. Fig 4: Sustainability of EQUIST Regarding the actual costs of the trainings and application of EQUIST and related equity based planning approach, few individuals were able to speak to these as a limited number of organizers of EQUIST training were interviewed in the assessment. With regard to the likelihood of the EQUIST s continued or repeated use without ongoing UNICEF support, the majority (91%) agreed that this was unlikely at current levels of support and engagement, and that its use would dwindle. It was suggested to increase the buy in for the EQUIST tool within several quarters e.g. National planning authorities, Ministries of Health and Bureaus of Statistics. At Makerere University in Kampala, a proposal was made to include a course model on EQUIST and also build a community of practice to ensure the institutionalization of the EQUIST. It was recommended for the World Health Organization to also buy into the tool. But there was also a proposal that the use of EQUIST without UNICEF support was possible if the strategies for updating the data were totally owned by countries and if countries were enabled and given full access to partake in this process. The majority, (70%) of respondents, agreed that there was limited systematic documentation and dissemination of good and bad practices or lessons learned on EQUIST both at the global and the national level, which has limited the enhancement of EQUIST implementation. There is need for more systematized engagement of UNICEF Headquarters with Country Offices to allow constant feedback and enhancement where needed. A proposed approach is to have a community of practice for the countries that are already using the tool and also webinars on the tool targeting learners. Some evidence indicates that EQUIST has not fully supported Country Offices in their advocacy among counterparts and partners for equity-focused programming. To further strengthen UNICEF s approaches to health sector planning, priority-setting and resource allocation to increase equity, technical assistance as well as a multi-sectoral approach and partnerships are critical. 11

20 4. Conclusions The aim of this assignment was to assess the use of EQUIST for strategic planning and prioritization of MNCH interventions and allocation of related resources. In essence, the tool and approach, scored more favorably in terms of the relevance of EQUIST to country policies, processes and procedures in health sector planning and priority-setting. There is momentum on the use of EQUIST in several contexts and the evidence suggests that EQUIST has played a visible and discernible role in strategic planning, prioritization and advocacy related to programs for marginalized groups in several countries. However, the tool has been less visible in informing resource allocation as other tools, such as the OneHealth and the MBB have been viewed as more helpful. In terms of implementation effectiveness, there is need for more timely and adequate support provided by UNICEF to country teams. Lastly, in terms of sustainability, given the current overwhelmingly unilateral support from UNICEF to sustain EQUIST, the long-term sustainability of EQUIST will benefit from UNICEF s ability to increase buy-in from other UN agencies or organizations. It would also be useful to document key lessons for effective implementation of EQUIST and follow up. For the countries that have embraced EQUIST, there is a strong likelihood of future tangible MNCH outputs and emerging outcomes resulting from the use of the tool. However, there is need to further evaluate this. Furthermore, the equity-based approach is paramount to UNICEF's programming. It is critical not to equate a tool to an approach as one can still have an equityapproach without necessarily using EQUIST, but it is a useful tool for identifying marginalized populations, and for other dimensions such as impact and related cost in terms of lives saved. The performance on the key indicators are summarized in the chart below. Fig 5: Summary across all indicators 12

21 5. Recommendations The results of this assessment outline strengths and weakness of EQUIST regarding relevance, training, implementation effectiveness, and sustainability. Based on these results, the following recommendations may improve the quality of EQUIST and its implementation: 1) EQUIST training - As teams were trained recently, there is need for more TA and support strengthening for enhanced use of the tool - Furthermore, EQUIST capacity has primarily been built within the UN system, hence, it would be helpful to scale up EQUIST beyond this, particularly to the governments, civil society and academia in recipient countries. - Improve the quality of the training for example by increasing the training depth and duration (to at least 5 days), use of multiple facilitators and leveraging on local TA in country. More time should be dedicated to scenario development & costing module. - Facilitate continuous online learning through webinars, step by step videos and a communities of practice. - Provide regular updates on EQUIST by the core team at UNICEF or a future lead agency - Update training tools and manuals - Build the capacity of EQUIST at the sub-national level - Improve the timeliness of the support provided on EQUIST 2) Data and Documentation - Improve in the availability and quality of EQUIST data- including comprehensive sub national data system to feed into EQUIST. - Provide local data access to users to regularly update their data, build/embed the equitybased dashboards/ visualizations within the country contextualized national Routine Health Information System (RiHIS), and georeferenced data. - Systematically document the good and bad practices around the use of EQUIST in order to enhance EQUIST implementation - Link the tool to implementation - through publishing papers using EQUIST analysis. - The option to link data (e.g multiple deprivation) has enormous benefits. Multiple deprivation is a relative measure of multiple deprivation expressed at small area level and covering an entire country 3. Domains may include; income and material deprivation, employment deprivation, education deprivation, health deprivation and living environment deprivation domains. Include multiple deprivation features so different indicators can be overlapped to show different dimensions of deprivation of outcomes, e.g. map/chart showing deprivation due to combined indices of under 5 mortality, stunting, wasting or combining status of ante natal care, skilled birth attendance and immunization coverage, etc. 3) EQUIST Platform - Recommended improvements - Align EQUIST with other similar tools such as the MBB, One Health Tool - Improve in the simplistic bottleneck/strategy approach - Enhance EQUIST s costing abilities - Display of costing prior to running LiST so scenario analysis could be less trial and error with respect to budgeting. Reviewing the costing data which seems high per capita 3 David McLennan, 2012, Indices of Deprivation, Mapping the spatial distribution of multiple deprivation at small area level and their uses for targeting area-based regeneration policies, et al; Institute for Clinical Systems Improvement. Achieving Accountability for Health and Health Care. Environment and Planning A 2006, volume 38, pages

22 - Include other thematic focal areas for example; adolescent health and nutrition - Include multiple deprivation features so that different indicators can be overlapped to show different dimensions of deprivation of outcomes, e.g. map/chart showing deprivation due to combine indices of U5MR, stunting, wasting or combining status of ANC, skilled birth attendance and immunization coverage, etc. - Improve internet connectivity for users- perhaps design offline version - Allow the capability for countries to routinely update their data 4) Sustainability of EQUIST - Strengthen efforts to advocate for and market the Tool both at country and at global levels. Country leadership is critical for this to happen. It would be helpful to have country champions to facilitate policy dialogue on the use of EQUIST. Consider promoting experience-sharing among countries. As part of advocacy efforts, demonstrate the usefulness of EQUIST for RMNCH investment cases and consider introducing requirements for country offices to use EQUIST data in the preparation of the CPDs. - Consider broadening EQUIST s areas of focus as the health sector contributes to only 20% of the population health; the rest is 80% are determined by socio-economic factors, health behavior and physical environment. These areas, if integrated with EQUIST, will make the tool stronger. - Explore the opportunity for ongoing assessments and the potential for further evaluation of EQUIST outcomes. 15

23 6. Annexes 6.1 Country applications CASE 1: Strengthening Capacity in Equity-informed Health and Nutrition Planning and Service Delivery in Sudan Background: Sudan, was not able to achieve the MDGs set for 2015, to eradicate poverty, reduce under five (U5) and MMR and halt the impacts of Malaria, TB and HIV/AIDS along with other MDGs targets that were strongly interlinked with the direct health indicators. In view of the above, without greater efforts to utilize the limited resources available from different stakeholders and the government using an equity focus approach, achieving Sustainable Development Goals (SDGs) by 2030 will remain distant. Relevance: The FMOH has led the third health sector strategic plan (HSSP) as part of the 25 years National strategy, with participation of all stakeholders. The aim of the strategy is to improve the health of Sudanese people with focus on the poorest and most at health risks of the population. The strategy is articulated around three main goals: 1) Strengthen and expand PHC services 2) Improve access to quality of health services and 3) Protect population specially the poor against catastrophic expenditure on health through expansion of health insurance. One of the key 10 strategic objectives of the RMNCAH strategic plan is achieving universal health coverage by RMNCAH packages through accelerating coverage of high impact interventions, linking child survival to other inventions such as reproductive health, family planning, maternal health, targeting most disadvantaged areas. Prioritization of interventions is key to the implementation arrangement of the 10 in 5 strategy framework. In this regards, EQUIST provides a quick glance at most in need settings, along with providing estimates for cost of selected interventions, helping national programmes in prioritization of interventions and geographical areas. Training: UNICEF organized a 3 day workshop, a total of 28 senior programme managers from the government and a group of stakeholders from the UN sister agencies and few international NGOs such as Italian Cooperation for Development on EQUIST. The workshop was facilitated by UNICEF staff from HQ, MENA Regional Office along with Country Office (CO) staff. Participants were divided into four groups: EPI, maternal and newborn, nutrition and health of under-five. Data from SHHS 2010 and MICS 2014 was included in EQUIST. Data is available from S3M 2013, although it precedes MICS 2014 but it provides data from locality level. Old data of DHS 1990 is available, yet it s very old but will be useful for advocacy, as many critical indicators for nutrition and sanitation are still stagnant around that level (see country context above). From day 2-3, each group started to work on thematic areas described above, using the seven steps, presenting findings and further move to other steps. Main outcomes of exercise: On day three, participants consolidated the findings including costing of selected interventions. Findings were discussed in plenary. Discussion on gaps in data took place, as participants are progressing in building scenarios. Each group was able to define priority underlying causes through group brain storming and reasoning. This was followed by navigating using indicators through different priority geographical areas as per EQUIST stepwise approach. Scenarios were costed per total lives saved and by per capita cost. The lessons learned following the plenary discussion was that some groups had inflated rates in some costs, due to incompleteness of Sudan s data. It was evident to the participants that validation of data and group discussions to agree on what interventions knowing the country context are key important factors that makes EQUIST an efficient tool. Recommendations: EQUIST is useful for programmes of child survival. There is a need to add other social indicators especially cultural factors that underline many causes for maternal, newborn and child deaths/poor health. Updating the tool and building national capacity on how to update it. Use other disaggregated data by locality (S3M etc..). Aligning EQUIST with other initiatives such as DHSS and within routine information of programme areas. Cascade of the training. Advocacy among other partners who were not able to participate in the workshop. 16

24 CASE 2: The application of EQUIST in Cameroon s Global Finance Facility (GFF) investment case development in Cameroon Background: In 2017, UNICEF organized an EQUIST workshop in Cameroon which targeted senior programme managers from the government, UN agencies and international NGOs. All were key for reproductive, maternal, newborn, child and adolescent health (RMNCAH) programming. Despite progress in the health sector, some health indicators in Cameroon have worsened. For example, maternal mortality increased from 430 per 100,000 live births in 1998 to 782 per 100,000 in There is also a geographical disparity in health indicators and access to health services in the country, particularly between the 3 northern regions and the rest of the country. Infant and child mortality remains high in poor parts of the country. Relevance: Equipped with technical skills on the use of the tool, the team conducted a situational analysis which shown that regional disparities had increased over the last ten years. The regions of Adamaoua, the East, the Far North and the North were those where the situation remained the most critical. To guarantee an equity perspective and focus on the most disadvantaged, these regions were prioritized for the key RMNCH interventions. In using the tool, the team completed EQUIST s sevenstep theory of change approach. Bottlenecks preventing effective coverage of the population with high-impact interventions were identified on the demand side as well as the supply side. Costs, acceptability and continuity were the main bottlenecks for the various identified high impact interventions. Several workshops allowed to analyze the different epidemiological conditions utilizing EQUIST. Priority areas were chosen on a consensual basis among the key stakeholders, while taking into consideration data on the respective burden of disease and levels of coverage through high-impact interventions and bilateral and multilateral consultations with key stakeholders. Main outcomes of the exercise: In line with sustainable development objectives and taking into account the need to move towards universal access to care, the main objective of Cameroon s investment dossier is to reduce maternal, neonatal, infant mortality / morbidity, and to promote the reproductive health of adolescents in the Adamaoua, Northern, Far North and East Cameroon regions. The areas of intervention that were selected for the investment case after the prioritization process were: combating maternal and neonatal mortality, malnutrition, adolescent reproductive health, and vital statistics registration and statistics. For each of these areas, a series of high impact interventions were identified using the EQUIST tool based on their respective potential impacts in terms of lives saved. These include: Immunization, family planning, prenatal care, postnatal care, assisted childbirth infants, IMCI, antibiotic therapy, and the importance of girls access to education. Limitations: Despite the fact that the EQUIST tool did not take into account the specificity of adolescents / young people, given their increased vulnerability or RH problems and their relatively lower coverage by some high-impact interventions, it is imperative to take specific measures in a rational manner to improve their coverage in all priority regions. Although not in the priority areas, Douala and Yaoundé were particularly targeted because of the increasing adolescent population in these cities due to the massive rural exodus. 17

25 6.2 Theory of Change 19

26 6.3 Evaluation Matrix OECD DAC CRITERIA Relevance Implementation Effectiveness QUESTIONS AND SUB-QUESTIONS 1. How relevant were EQUIST to country policies, procedures and processes in health sector planning, priority-setting and resource allocation? 2. To what extent does the context in which EQUIST is used determine its relevance and/or appropriateness? Are there factors or conditions that provide more or less suitable settings for its use? 3. How does the equity-focused approach using EQUIST compare to existing or alternative means of health sector planning, priority-setting and resource decisionmaking? Are there complementarities between EQUIST and other tools or approaches? 4. How adequate is UNICEF s engagement with governments and coordination with other partners to introduce and advocate use of EQUIST? What partnership modalities have been most constructive? 5. How adequately has UNICEF documented and disseminated information on EQUIST to potential users? How easy or difficult was it to explain the application of the tool, and was the training on its use felt to be adequate for its ongoing use? Or was follow up oversight required? How accurate and adequate was the database for the application of the tool? How well did trainees and counterparts manage the application of the tool in the scenario analysis, steps 4-6? What have been the steps and costs associated with development and launching of the EQUIST tool? What are the costs per country application? To what extent does EQUIST provide timely inputs to policy and programme decision-making? How could timeliness (i.e. on-time delivery) be improved or optimized? To what extent has EQUIST contributed to/resulted in the identification of new and/or refined target populations, health conditions, interventions and understanding of bottlenecks. To what extent have stated country priorities been adapted to account for the contributions of EQUIST? To what extent has allocation of resources been increased or rationalized in the direction of an increased equity focus, based on the tool? What is the likely sustainability of the changes in priority and resource allocation induced by use of the tool? To what degree has EQUIST s leverage relied on donor or partner funding? Has it resulted in reallocation of public sector resources in a sustainable way? Are there specific populations, health conditions and/or interventions which have been particularly affected by EQUIST use? To what extent has EQUIST contributed to capacity-building for planning, prioritysetting and resource allocation decisions? What are the key factors and conditions associated with effective use of the tool? How do those factors and conditions vary across context? 20

27 Equity and Sustainability What lessons can be drawn to strengthen UNICEF s approaches to health sector planning, priority-setting and resource allocation to increase equity? To what extent does EQUIST improve and/or strengthen decision-makers focus on the most marginalized? To what extent is EQUIST s role visible or discernible in monitoring and accountability mechanisms related to programs for marginalized groups? To what extent has EQUIST supported UNICEF COs in their advocacy among counterparts and partners for equity-focused programming? To what extent have sustainability considerations (technical, financial, institutional) been integrated in EQUIST design and implementation? What is the likelihood of the tool s continued or repeated use without UNICEF support? How adequate are UNICEF s actions in reinforcing EQUIST with respect to direct support, upstream work and creation of enabling environments needed for sustainability? 21

28 6.4 Interview and Discussion Guide Background In support of the SDGs, UNICEF has an agency-wide Strategy for Health, The Strategy envisions a world where no child dies from a preventable cause, and all children reach their full potential in health and well-being. In order to achieve this, all programmes supported by UNICEF aim to: address inequities in health outcomes, promote integrated, multi-sectoral policies and action and strengthen health systems, with a particular focus on emergency preparedness, response and resilience. To facilitate country processes, the Bill and Melinda Gates Foundation (BMGF) funded the UNICEF Health Program through the U.S. Fund for UNICEF to develop the EQUIST tool. EQUIST identifies cost-effective interventions, prioritizes key bottlenecks that constrain their coverage and targets the most effective and equity-focused strategies, that can address these bottlenecks to increase MNCH coverage and fast track the reduction of maternal and child mortality, especially among the deprived populations. At the country level, UNICEF will work with governments to adapt and apply the EQUIST as an evidence-based and equity-focused programming tool to sharpen country strategies and plans that aim to improve the coverage of high impact MNCH and nutrition interventions. The application of EQUIST will involve: Building Capacity: developing the capacity of technical and managerial cadres within UNICEF Country Offices, government officials and key national partners; Conducting Scenario Analyses: supporting stakeholders in identifying priority populations, interventions, bottlenecks and operational strategies and estimating the required additional resources; Facilitating Dialogue: facilitating policy dialogue among key decision makers; and Documenting Lessons Learned: further support to capitalize on the lessons learned in the process through: 1) systematic documentation; and 2) peer-to-peer experience sharing (e.g. community of practice, learning tours, coaching and mentorship). As a result of these activities, countries have the opportunity to: 1. Develop evidence-based, equity-focused strategies and sharpened plans for MNCH and nutrition. 2. Mobilize resources and re-allocate support to address remaining gaps in strategic plans; and 3. Institutionalize an equity-based approach to planning and HSS. The primary purpose of the interview is to examine UNICEF s equity-based approach and web-based tool, EQUIST, for strategic planning and prioritization of maternal, newborn and child health (MNCH) interventions and allocation of related resources for health systems strengthening (HSS). Objectives of the Assessment The objective of the interview is gather information to answer the following broad questions: Have the objectives of the equity-based approach and use of EQUIST been met (e.g. building capacity, developing equity-based strategies / plans; mobilizing resources; and institutionalizing equity-based approaches) and what has been achieved so far? What factors have contributed to success or failure of the policy dialogue and use of EQUIST what is working or not working? Results will be used to develop findings for a formative assessment of facilitated equity-focused policy dialogue and the use of the EQUIST tool. 22

29 ANNEX 2: INTERVIEW GUIDE - UN AND MOH PERSONNEL Contact Information (in case more information is needed) Name Organization Position / Department Country Part A: TRAINING 1. Did you participate in a training on the use of EQUIST or an equity-based approach to planning? >>> IF NO, skip to next section 2. Please describe the process and outcome? [ENTRY BOX] 3. What month and year did the training take place? 4. How adequate was the training to understand EQUIST and the equity-based approach to strategic planning? Please describe. [ENTRY BOX] 5. How adequate were the distributed training materials, documentation and manuals? Please describe. [ENTRY BOX] 6. Has the EQUIST training contributed to capacity building for planning, priority-setting and resource allocation decisions? YES / NO If Yes, please describe [ENTRY BOX] 7. Do you have any (additional) recommendations for improvement to the training sessions and training materials? [ENTRY BOX] Part B: RELEVANCE 6. How relevant is EQUIST to country processes in health sector planning, priority-setting and resource allocation? [ENTRY BOX] 7. Does EQUIST improve and/or strengthen decision-makers focus on the most marginalized? [ENTRY BOX] 8. How does EQUIST compare to existing or alternative means of health sector planning, priority-setting and resource decision-making? [ENTRY BOX] 9. Have you been motivated towards the equity-based approach to planning and the use of the EQUIST tool? [ENTRY BOX] 10. Do you have any (additional) recommendations for improving the EQUIST tool or the equitybased approach to planning? Part C: IMPLEMENTATION EFFECTIVENESS 11. To develop a national or district-level strategic plan, have you ever used the EQUIST tool or related equity-based approach? If so, please describe the process and outcome? YES / NO a) If yes, please describe the process used and your role [ENTRY BOX] b) >> No [SKIP to Next Section] 23

30 12. Were any changes made to national, district or partner plans as a result of the use of EQUIST or an equity-based approach to planning? Please describe. c) If Yes, please describe below [ENTRY BOX] d) If No, why not? [ENTRY BOX] 13. Has EQUIST provide timely inputs to policy and programme decision-making? YES / NO e) If Yes, please describe. [ENTRY BOX] f) If No, How could timeliness (i.e. on-time delivery) be improved or optimized? [ENTRY BOX] 14. Has the equity-based planning approach or EQUIST contributed to capacity building for planning, priority-setting and resource allocation decisions? YES / NO If Yes, please describe [ENTRY BOX] 15. What are the key factors and conditions associated with effective use of EQUIST and the equity-based approach to planning? [ENTRY BOX] 16. Has UNICEF provided adequate and timely to support for the use of EQUIST or an equitybased planning approach? Please describe. [ENTRY BOX] 17. Have you been motivated towards the equity-based approach to planning and the use of the EQUIST tool? [ENTRY BOX] Part D: EQUITY AND SUSTAINABILITY 18. Have sustainability considerations (technical, financial, institutional) been integrated into implementation of the equity-based planning approach and use of EQUIST? Please describe [ENTRY BOX] 19. What is the likelihood of the EQUIST s continued or repeated use without UNICEF support? [ENTRY BOX] 20. To what extent have good and bad practices or lessons learned around EQUIST been systematically documented and disseminated to enhance EQUIST implementation? [ENTRY BOX] 21. Have additional/new financial (domestic or external) and human resources been committed to the use of EQUIST or the equity-based approach to planning? Please provide examples. 22. Has EQUIST been scaled up with alternative resources and programming support? Please provide examples. [ENTRY BOX] Part E: FACILITATION 23. Did you facilitate or organize the training or use of EQUIST amongst other organizations? >>> If NO, then END survey 24. What organizations were involved in the training or use of EQUIST? 25. Approximately how many people were involved in the training or use of EQUIST? 26. What were the approximate costs involved in the country application of EQUIST and related equity-based planning approach? 24

31 27. What lessons can be drawn to strengthen UNICEF s approaches to health sector planning, priority-setting and resource allocation to increase equity? 28. Is EQUIST s role visible or discernible in monitoring and accountability mechanisms related to programs for marginalized groups? 29. Has EQUIST supported UNICEF COs in their advocacy among counterparts and partners for equity-focused programming? 25

32 6.5 Online survey tool 26