Ministerial Forum for Health Ministers Harvard University June 6, 6 In association with Big Win Philanthropy and with the support of the Bill and Melinda Gates Foundation EVALUATION REPORT In association with Big Win Philanthropy and with the support of the Bill and Melinda Gates Foundation Analysis and compilation by Helene Perold and Marinda Weideman with Jacques P. Perold for Helene Perold & Associates
Contents. Introduction.... Overview of evaluation findings.... Evaluation methodology.... Competency profile assessment.... Changes in group feedback on self-reported competency levels... 6. Pre- and post-forum impact: Group Net Confidence Ratings... 8. Changes in individual feedback on self-reported competency levels... Country... Error! Bookmark not defined. Country... Error! Bookmark not defined. Country... Error! Bookmark not defined. Country... Error! Bookmark not defined. Country... Error! Bookmark not defined. Country 6... Error! Bookmark not defined. Country 7... Error! Bookmark not defined. Country 8... Error! Bookmark not defined. Country... Error! Bookmark not defined. What is likely to change?... 7. Impact on the ministers leadership role... 7. Impact on the health minister s job... 9. Challenges envisaged.... Prospects for collaboration with the finance minister.... Other feedback... Annexure Ministerial Forum for Health Ministers Program... Annexure Day Feedback Report... 7 Annexure Day Feedback Report... Annexure Day Feedback Report... 6
. Introduction The annual Ministerial Forum for Health Ministers is a core part of the Ministerial Leadership in Health Program at Harvard University. The main goals of the Forum are to increase ministerial and political leadership effectiveness in the effort to improve health outcomes in developing and emergent countries by increasing efficiency in the use of national budgets and delivery of public health services, and establishing better standards of care. Ten health ministers attended the 6 Health Ministers Forum from June - 6, 6. The ministers came from the Afghanistan, Botswana, Burkina Faso, Guatemala, Guyana, Liberia, Niger, Nigeria, Sierra Leone and Country. A real-time evaluation process was conducted over the five days of the Forum in order to assess the extent to which participants felt they had gained useful knowledge and skills, and how well prepared they felt to implement these skills after they returned home. This report outlines the learnings recorded by each of the participants over the course of the five days. It documents how they rated changes in their knowledge and confidence levels, and reflects their self-assessed competence to support health sector strengthening.. Overview of evaluation findings The 6 Ministerial Forum for health ministers achieved an extraordinary rate of success, as measured by the changes in knowledge and confidence levels recorded by nine out of ten ministers who completed the competency assessment at the start and end of the five-day process. By the end of the week, the ministers showed a % Net Confidence Rating across all three components of the Forum: transformational leadership, health financing, and organizing for implementation. These self-assessed results suggest that the concepts and practical focus of the Forum substantially enhanced their knowledge and confidence levels in respect of ministerial competencies that were measured: The ministers became convinced of the transformative value of their leadership in strengthening their health systems, and the % net confidence ratings indicate they feel well-equipped to carry this out. Health financing was the area in which the ministers recorded their steepest learning curve, with confidence levels rising by 89% in respect of understanding how to increase health budget effectiveness (from % pre-forum to % post-forum). In the delivery sphere, the ministers recorded learning gains of 78% in respect of knowing how to plan, implement and monitor the delivery of their health system strengthening priorities (from % pre-forum to % post-forum). Figure depicts pre- and Post-Forum net confidence ratings for all twelve competencies assessed. One minister did not complete the end-line assessment and his ratings could thus not be included in this overall analysis. His daily feedback is, however, recorded in Annexures, and. Net Confidence Ratings show the percentage of participants who, by the end of the Forum, recorded high levels of knowledge and confidence, and felt well equipped to apply different competencies associated with the role of Health Minister. Given the small number of respondents (9 in this case), the net confidence increases should be regarded as indicative of the impact of the Forum.
This is the first year that the Forum has been able to show % self-assessed net confidence ratings across all aspects of its content. Although the precise reasons for these high ratings could not be determined from the data, the overall sense that ministers found the Forum a highly valuable exercise was also reflected in the exit interviews conducted at the end of the Forum. The higher the usual ratings may also reflect the the continuous refinement of the content and delivery of the Forum curriculum, as well as the possibility that the selection of participants for this Forum produced a particularly dynamic group that was able to respond positively to all aspects of the training and discussion. Figure : Post-Forum Net Confidence Ratings (n=9) Figure illustrates the changes in the group competence profile and shows the five areas in which the group demonstrated the steepest learning curves. Four of these instances relate to health financing strategies and one concerns planning, implementing and delivering priorities for strengthening the health system. The health financing strategies which drew the lowest ratings from the ministers at the start of the Forum and subsequently recorded the largest increases in confidence levels were: Increase value for money in public health; Plan to reduce waste, and increase efficiency in procurement and supply management, and improve health worker productivity; Identify fiscal space in the health budget; Increase health budget effectiveness. Participants were asked to rate the extent to which they felt equipped for their role as health minister in respect of competencies. A five-point rating scale was used, from ( not at all equipped ) to ( fully equipped ).
Figure : Pre- and Post-Forum competence profile: Group average (n=9) During the five exit interviews conducted at the end of the Forum, some ministers commented that the Forum had effectively combined big picture thinking with practical ways in which they could achieve their legacy goals. These include defining a legacy, realizing this vision by prioritizing and building a support base, and being transparent. A number mentioned how closely the Forum program was aligned with their needs as health ministers. Consequently, they envisaged doing their jobs differently by focusing on their top priorities and building support for their priorities within their ministries and government more broadly; organizing their ministries more effectively; and monitoring progress more closely so as to speed up the reforms. Most of the health ministers who were interviewed were positive about the prospects for collaborating with their counterparts in finance. In two cases the health ministers already had good working relationships with their finance ministers, who were chairing important decision-making structures, which could be advantageous to strengthening the health sector. Another minister indicated that he would seek to work closely not only with his finance minister, but also with other ministers located in the office of the President and the office of the Prime Minister, because all these ministers will have to be on board for him to succeed with his planned reforms. It is important to note that although the results show that the health ministers started the week with very low levels of confidence about health financing, and ended feeling fully equipped to deal with these issues, only one of the ministers specifically cited efficiency and effectiveness gains as one of the changes she wishes to make on returning to her country. Three key challenges were mentioned by the five ministers who were interviewed: encountering resistance to change from colleagues in government and in their ministries; generating the data
required for evidence-based decision-making; and improving human resources for health, both in relation to the quantity of health workers and the quality of their work. Nevertheless the ministers felt well-equipped to tackle these challenges by using tools and approaches discussed at the Forum.. Evaluation methodology As was the case in previous years, the evaluation process was designed to track the development of participants knowledge, skills and confidence, and to provide the Harvard team with daily feedback about how the participants responded to the program content, presentations and discussion. Specific aims of the evaluation process were as follows: a. Provide comparative (quantitative) self-reported data that measures what the ministers learned by the end of the Forum and how equipped they felt to take action in the health sector; b. Provide the Harvard team with daily feedback from the ministers on the value of what they learned each day; c. Conduct qualitative interviews to gain closer insight into the quantitative scores provided by the ministers. The indicators for these assessments reflect the content of the Forum and are listed in Section below, as well as in the daily reports in Annexures, and. Three instruments were used to collect quantitative and qualitative information about participants perspectives on different parts of the program and whether they learnt anything new: A competency profile was designed to produce baseline scores for self-reported impact of the workshop. At the beginning of the Forum ministers were asked to rate the extent to which they felt equipped to carry out competencies required to strengthen the public health sector. At the end of the Forum the same instrument was administered to capture the final scores needed to assess what the ministers felt they had gained. These results were compared with the ratings they recorded at the start of the Forum. A daily assessment instrument was used to capture the extent to which the ministers felt they had improved their knowledge, confidence and skill in relation to the issues covered on that day. This produced information that helped presenters take the temperature of the Forum on a daily basis, and indicates which sessions were most and least successful for the participants. An interview schedule was developed to guide the interviews conducted with five ministers at the end of the Forum. All ten participants completed the baseline competency assessment, but one did not complete the final assessment. The comparison of scores therefore reflects only the scores provided by the nine participants that completed both. With the exception of one participant on day three, all participants completed the daily feedback (see Annexures, and ). Data analysis involved analysing the quantitative data recorded on the different instruments, and documenting the qualitative feedback received on different days. The quantitative data is presented in two ways:
A competency profile was developed for individual participants and for the group as a whole, showing the changes in ratings that signify how well equipped the ministers felt to implement the actions covered during the sessions. These findings are described in spider diagrams. A Net Confidence Rating (NCR) was computed to measure the spread between respondents who felt well-equipped or ill-equipped about a range of competencies, before and after the Forum. The net confidence rating illustrates the concentration of positive and negative responses for each action. Neutral responses are not taken into account since they do not add meaningful information to the analysis. A negative NCR indicates that more respondents felt ill-equipped than those who felt well-equipped. Given the small number of respondents (nine in this case), the net confidence increases should be regarded as indicative of the impact of the Forum.. Competency profile assessment All ten ministers completed the baseline assessment called a competency profile on the first day of the Forum, while nine completed the end-line assessment. The responses from the minister who did not complete the end-line assessment are therefore not included in the comparative data, but are included in the daily assessments (see annexures). The assessment consisted of a set of twelve actions related to the Forum content on the financing of health and social sector programs. Ministers used a -point scale to indicate the extent to which they felt equipped to carry out the competencies listed below. The scale used by participants ran from ( not at all equipped ) to ( very well equipped ). More specifically the five options were, not at all equipped, poorly equipped, somewhat equipped, equipped, and fully equipped.. Define my leadership goals to establish a transformational legacy during my term in office.. Understand how to assert effective leadership in building support for my policy goals.. Identify core priorities for accomplishing my legacy.. Understand opportunities to increase health budget effectiveness.. Identify fiscal space in the health budget through reallocation, saving and additional revenues. 6. Identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment. 7. Plan approaches for reduced waste and increased efficiency in procurement and supply management, and improved health worker productivity. 8. Plan, implement and monitor delivery on health system strengthening priorities. 9. Organize an effective ministerial implementation team to deliver health sector strengthening priorities.. Increase collaboration with the Finance Minister to strengthen health sector performance.. Use data to monitor health delivery system performance and implementation of policy priorities.. Hold officials accountable for performance, delivery and outcomes. A comparison of the scores provided at the baseline and follow-up assessment is provided below.
. Changes in group feedback on self-reported competency levels The section compares the ratings on the set of twelve indicators as provided at the start and close of the Forum. The aim is to identify shifts in reported levels of knowledge and confidence over the course of the Forum. There was a positive shift in levels of confidence from the baseline to the final assessment for all twelve indicators measured. The average score across all indicators increased from.6 to.7 (an increase of. on the -point scale). The range of scores increased from between. to. in the baseline assessment, to between.. and.8 in the final assessment. Figure shows the average group scores for the baseline (blue) and final (orange) for the nine ministers who completed both assessments. Figure : Comparison of average group scores on the Competency Assessment Competency self-assessment: Baseline and final comparison Baseline Final a. Define my leadership goals to establish a transformational legacy i. Hold officials accountable for performance, delivery and b. Understand how to assert effective leadership in building support for my k. Use data to monitor health delivery system performance and j. Increase collaboration with the Finance Minister to strengthen health sector c. Identify core priorities for accomplishing my legacy. d. Understand opportunities to increase health budget effectiveness. i. Organize an effective ministerial implementation team to deliver health e. Identify fiscal space in the health budget through reallocation, saving and h. Plan, implement and monitor to deliver on health system strengthening g. Plan approaches for reduced waste and increased efficiency in procurement As noted, the ministers reported increased levels of confidence for all twelve of the competency indicators listed. The list below demonstrates in descending order the extent (on the point scale) of self-assessed increases in confidence. Table : Changes in group confidence levels by point and range increases f. Identify opportunities to increase efficiency in budget and health resource Indicator Identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment Plan approaches for reduced waste and increased efficiency in procurement and supply management, and improved health worker productivity. Point increase Score range...6...7 6
Indicator Point increase Score range Plan, implement and monitor delivery on health system strengthening priorities....7 Identify fiscal space in the health budget through reallocation, saving and additional revenues....6 Understand opportunities to increase health budget effectiveness.... Use data to monitor health delivery system performance and implementation of policy priorities. Organize an effective ministerial implementation team to deliver health sector strengthening priorities..7.7.9.9.8 Understand how to assert effective leadership in building support for my policy goals..8.7. Increase collaboration with the Finance Minister to strengthen health sector performance..8.8.6 Hold officials accountable for performance, delivery and outcomes..8..8 Define my leadership goals to establish a transformational legacy during my term in office..7..8 Identify core priorities for accomplishing my legacy....8 The first three in the list indicate the performance areas for which the largest increases in confidence were reported, and the last three, the performance areas for which smaller increases in confidence were reported. Figure shows the change between the average baseline and final scores per indicator. Figure : Average group scores showing confidence ratings before and after the Forum Competency self assessment: comparison between baseline and final Hold officials accountable for performance, delivery and Use data to monitor health delivery system performance Increase collaboration with the Finance Minister to Organize an effective ministerial implementation team to Plan, implement and monitor to deliver on health system Plan approaches for reduced waste and increased Identify opportunities to increase efficiency in budget and Identify fiscal space in the health budget through Understand opportunities to increase health budget Identify core priorities for accomplishing my legacy. Understand how to assert effective leadership in building Define my leadership goals to establish a,8,7,7,8,6,9,8,,7,,7,,6,,6,,,,8,7,,,8 6 Baseline Final 7
By the end of the Forum, all competence indicators received scores of. (out of ) or higher, indicating high levels of confidence and showing that, as a group, the ministers felt equipped to fully equipped in all action areas measured.. Pre- and post-forum impact: Group Net Confidence Ratings Another way of viewing the impact of the Forum on the health ministers, is to look at how the group s net confidence levels changed from feeling poorly equipped at baseline, to feeling fully-equipped at end-line. Although the analysis reflects a very small number of ministers (n=9), the trends evident in this analysis suggest that the Forum was highly effective in producing high levels of knowledge and confidence in all 9 ministers who completed the baseline and final competency assessments. Figures -7 illustrate these changes across the three key themes: transformative leadership, health financing, and organizing the ministry for delivery on policy goals. Transformative leadership: This was the theme about which the participants felt most confident at the start of the Forum. As shown in Figure, 7 out of 9 ministers started out feeling equipped to define their leadership goals for leaving a transformational legacy; 8 out of 9 felt equipped to identify core priorities for accomplishing their legacy; but approximately half the ministers ( out of 9) were ambivalent about asserting effective leadership to build support for their policy goals. By the end of the Forum, all 9 ministers indicated that they felt equipped or well-equipped to use these competencies. The graph below indicates that the Forum proceedings produced self-assessed confidence ratings of % across all three ministerial leadership competencies. Figure : Transformative Leadership - net confidence ratings (n-9) 8
Health financing: This was the component of the Forum in which the health ministers recorded the highest increases in learning and confidence. At the start of the Forum more than half the ministers indicated that they felt ambivalent or poorly equipped to carry out five health financing actions: (a) increase health budget effectiveness; (b) identify fiscal space in the health budget; (c) increase efficiency in health budget and resource utilization; (d) reduce waste, increase efficiency in procurement and supply management, and improve health worker productivity; and (e) increase collaboration with the finance minister to strengthen health sector performance. As shown in Figure 6, net confidence ratings at the start of the Forum ranged between % and 6%. By the end of the Forum all the ministers indicated that they felt well-equipped to carry out these functions, producing a net confidence rating of % across all five indicators. The largest increase occurred in respect of understanding opportunities to increase the effectiveness of the health budget, with the confidence increases of 89% recorded across the group (from % at the start of the Forum to % by the end). Figure 6: Health financing net confidence ratings (n=9) Organizing for implementation: The ministers net confidence ratings show a wide range of entry points in respect of their confidence to use the delivery approach for policy implementation: Only out of 9 ministers indicated at the start of the Forum that they felt well-equipped to plan, implement and monitor the delivery of their priorities for strengthening their health system. On the other hand, 7 out of 9 felt well-equipped to hold officials accountable for performance, delivery and outcomes. Six out of 9 felt well-equipped 9
to organize an effective ministerial implementation team, and the same number were confident about using data to monitor health system performance. By the end of the Forum all 9 ministers recorded that they felt well-equipped to carry out each of these actions, which produced a % net confidence rating across the board. As shown in Figure 7, the sharpest increase in learning and confidence occurred in relation to planning, implementing and monitoring to deliver on the priorities for health system strengthening, with net confidence increases of 78% recorded across the group (from % at the start of the Forum to % by the end). Figure 7: Organizing for implementation net confidence ratings (n=9) These results indicate that the Forum produced important gains for the ministers. They show that the content, presentations and practical activities combined to move the group from low and midconfidence levels at the start, to % net confidence ratings by the end. The results may also signal that the selection of participants for this Forum produced a dynamic group that was able to respond positively to all aspects of the training and discussion. As is shown in Section below, some of the health ministers commented that the Forum content and process resonated closely with their own contexts and concerns. This may also help to explain the very high net confidence ratings produced over the five days.. Changes in individual feedback on self-reported competency levels This section documents the learning gains of individual participants at the Forum. The figures below compare the baseline (pre-forum) and final (post-forum) competency scores recorded by individual participants, and provide an indication of areas in which they grew in confidence.
Country This minister s average score (across all indicators) increased from. in the baseline survey to in the final survey (a point increase of.). The range increased from between and in the baseline assessment, to between and in the final assessment. There was an increase in self-assessed confidence for six of the indicators (see below), whilst the confidence levels for the other six indicators remained the same. All scores in the final assessments were s and s, indicating that the participant felt equipped to fully equipped to execute the actions measured. Figure 8: Comparison of individual competency scores Country Competency self-assessment: Baseline and final comparison for Country Baseline Final j. Increase collaboration with the Finance Minister i. Hold officials accountable for k. Use data to monitor health delivery system a. Define my leadership goals to establish a b. Understand how to assert effective leadership c. Identify core priorities for accomplishing my legacy. d. Understand opportunities to increase i. Organize an effective ministerial implementation e. Identify fiscal space in the health budget through The list of indicators below demonstrates the extent of self-assessed changes in confidence. The data are presented in descending order, so that the first six demonstrate the areas in which confidence increases were largest, and the last six demonstrate the areas for which confidence levels did not change. Table : Changes in confidence levels by point and range increases Country Indicator h. Plan, implement and monitor to deliver on f. Identify opportunities to increase efficiency in g. Plan approaches for reduced waste and Point increase Understand opportunities to increase health budget effectiveness Define my leadership goals to establish a transformational legacy during my term in office Identify fiscal space in the health budget through reallocation, saving and additional revenues Identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment Plan approaches for reduced waste and increased efficiency in procurement and supply management, and improved health worker productivity Plan, implement and monitor delivery on health system strengthening priorities Score range
Indicator Organize an effective ministerial implementation team to deliver health sector strengthening priorities Point increase Increase collaboration with the Finance Minister to strengthen health sector performance Use data to monitor health delivery system performance and implementation of policy priorities Hold officials accountable for performance, delivery and outcomes Understand how to assert effective leadership in building support for my policy goals Identify core priorities for accomplishing my legacy Score range Table shows that at the outset the Minister was most confident about identifying his core priorities for his legacy, building support for his policy goals and applying some of the best practices in operationalizing the envisaged heath system reforms. During the Forum his confidence about transformative leadership, health financing and planning increased across the remaining six indicators, reflecting that he felt equipped to use of the competencies to lead these changes and fully equipped to identify his core priorities. Country The average rating across all indicators for the minister from Country increased from. ( equipped ) to ( fully equipped ). The score range increased from between. and in the baseline assessment to (fully confident) in all action areas in the final assessment. This shows that the minister felt fully equipped to undertake all the action areas measured.
Figure 9: Comparison of individual competency scores Country Competency self-assessment: Baseline and final comparison for Country i. Hold officials accountable for performance, delivery and outcomes. Baseline Final a. Define my leadership goals to establish a transformational legacy b. Understand how to assert effective leadership in building support for my k. Use data to monitor health delivery system performance and j. Increase collaboration with the Finance Minister to strengthen health sector c. Identify core priorities for accomplishing my legacy. d. Understand opportunities to increase health budget effectiveness. i. Organize an effective ministerial implementation team to deliver health e. Identify fiscal space in the health budget through reallocation, saving and h. Plan, implement and monitor to deliver on health system strengthening g. Plan approaches for reduced waste and increased efficiency in procurement f. Identify opportunities to increase efficiency in budget and health resource The list of indicators below demonstrates the extent of the minister s self-assessed increases in confidence. The data are presented in descending order, so that the first four demonstrate the areas in which confidence increases were largest, while the last six demonstrate the indicators for which no change was reported. Table : Changes in confidence levels by point and range increases Country Indicator Point increase Identify core priorities for accomplishing my legacy Plan approaches for reduced waste and increased efficiency in procurement and supply management, and improved health worker productivity Use data to monitor health delivery system performance and implementation of policy priorities Plan, implement and monitor delivery on health system strengthening priorities Score range Define my leadership goals to establish a transformational legacy during my term in office.. Identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment Understand how to assert effective leadership in building support for my policy goals Understand opportunities to increase health budget effectiveness
Indicator Identify fiscal space in the health budget through reallocation, saving and additional revenues Organize an effective ministerial implementation team to deliver health sector strengthening priorities Increase collaboration with the Finance Minister to strengthen health sector performance Point increase Hold officials accountable for performance, delivery and outcomes Score range The data suggest that the minister had the biggest knowledge gains in the component on transformation leadership, prioritizing and improving efficiency. This was also the component for which the minister felt least equipped at the start of the Forum. Country The minster s average score (for all indicators) increased from.6 at the baseline assessment to. at the final assessment, suggesting that his confidence levels changed from low to high overall. The score range changed from between and at baseline, to to at the final assessment, which suggests that the minister now feels equipped to fully equipped to undertake the actions measured. As the figure below shows, the minister s confidence increased across of the indicators, and remained the same (at a rating of ) for the indicator Identify core priorities for accomplishing my legacy.
Figure : Comparison of individual competency scores Country Competency self-assessment: Baseline and final comparison for Country i. Hold officials accountable for performance, delivery and outcomes. Baseline Final a. Define my leadership goals to establish a transformational legacy b. Understand how to assert effective leadership in building support for my k. Use data to monitor health delivery system performance and j. Increase collaboration with the Finance Minister to strengthen health sector c. Identify core priorities for accomplishing my legacy. d. Understand opportunities to increase health budget effectiveness. i. Organize an effective ministerial implementation team to deliver health e. Identify fiscal space in the health budget through reallocation, saving and The list of indicators below demonstrates the extent of the minister s self-assessed increases in confidence. The data are presented in descending order, so that the first three demonstrate the areas in which confidence increases were largest, while the last four demonstrate the areas in which confidence increased less dramatically, or remained the same. For the first three indicators listed, the increases in confidence were large, from poorly equipped to fully equipped. Table : Changes in confidence levels by point and range increases Country Indicator h. Plan, implement and monitor to deliver on health system strengthening g. Plan approaches for reduced waste and increased efficiency in procurement Point increase Hold officials accountable for performance, delivery and outcomes Identify fiscal space in the health budget through reallocation, saving and additional revenues Organize an effective ministerial implementation team to deliver health sector strengthening priorities Define my leadership goals to establish a transformational legacy during my term in office Understand opportunities to increase health budget effectiveness Plan, implement and monitor delivery on health system strengthening priorities Use data to monitor health delivery system performance and implementation of policy priorities f. Identify opportunities to increase efficiency in budget and health resource Understand how to assert effective leadership in building support for my policy goals Score range
Indicator Increase collaboration with the Finance Minister to strengthen health sector performance Identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment Plan approaches for reduced waste and increased efficiency in procurement and supply management, and improved health worker productivity Point increase Identify core priorities for accomplishing my legacy Score range In relation to the other Forum participants, this minister recorded comparatively low levels of selfassessed competence at the beginning of the Forum. It is therefore interesting that he rated his greatest gains in confidence and learning in respect of organizing an effective ministerial team, holding officials accountable for performance, and identifying fiscal space in the health budget through reallocation, saving and additional revenues. Country The minister s average score for the indicators increased from. ( somewhat equipped ) at the baseline assessment to.6 (almost fully equipped ) at the final assessment. The score range increased from between and in the baseline assessment, to between and in the final assessment. The minister s confidence increased for out of action areas measured. For the indicator Use data to monitor health delivery system performance his confidence level remained at. 6
Figure : Comparison of individual competency scores Country Competency self-assessment: Baseline and final comparison for Country i. Hold officials accountable for performance, delivery and outcomes. Baseline Final a. Define my leadership goals to establish a transformational legacy b. Understand how to assert effective leadership in building support for my k. Use data to monitor health delivery system performance and j. Increase collaboration with the Finance Minister to strengthen health sector c. Identify core priorities for accomplishing my legacy. d. Understand opportunities to increase health budget effectiveness. i. Organize an effective ministerial implementation team to deliver health sector e. Identify fiscal space in the health budget through reallocation, saving and h. Plan, implement and monitor to deliver on health system strengthening g. Plan approaches for reduced waste and increased efficiency in procurement f. Identify opportunities to increase efficiency in budget and health resource The list of indicators below demonstrates the extent of the minister s self-assessed increases in confidence. The data are presented in descending order, so that the first six demonstrate the areas in which confidence increases were largest, while the last three demonstrate the areas in which the minister s confidence increased, but to a smaller extent. There was no change in the minister s confidence levels in respect of using data to monitor the performance of the health system. His rating for this item stayed at ( equipped ). The table shows that the minister recorded the largest increases in learning and confidence in respect of health financing, as well as planning, implementing and monitoring delivery on his priorities for strengthening the health system. Table : Changes in confidence levels by point and range increases Country Indicator Increase collaboration with the Finance Minister to strengthen health sector performance Point increase Plan, implement and monitor delivery on health system strengthening priorities Plan approaches for reduced waste and increased efficiency in procurement and supply management, and improved health worker productivity Identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment Score range 7
Indicator Identify fiscal space in the health budget through reallocation, saving and additional revenues Point increase Understand opportunities to increase health budget effectiveness Define my leadership goals to establish a transformational legacy during my term in office Understand how to assert effective leadership in building support for my policy goals Identify core priorities for accomplishing my legacy Organize an effective ministerial implementation team to deliver health sector strengthening priorities Hold officials accountable for performance, delivery and outcomes Use data to monitor health delivery system performance and implementation of policy priorities Score range Country The minister s average score increased from.9 in the baseline assessment to.8 in the final assessment, indicating that his overall confidence level increased from equipped to almost fully equipped. The minister s score range increased from between and at baseline, to between and at the final assessment (with of the indicators at, signalling fully equipped ). There were two indicators for which the minister reported no change in confidence, with both rated at or at the start and end of the Forum: Identify fiscal space in the health budget through reallocation, saving and additional revenues. Increase collaboration with the finance minister to strengthen health sector performance. There was also one indicator for which the minister reported a slight decline in confidence, from to : Understand opportunities to increase health budget effectiveness. 8
Figure : Comparison of individual competency scores Country Competency self-assessment: Baseline and final comparison for Country Baseline Final i. Hold officials accountable for a. Define my leadership goals to establish a b. Understand how to assert effective leadership k. Use data to monitor health delivery system j. Increase collaboration with the Finance Minister c. Identify core priorities for accomplishing my legacy. d. Understand opportunities to increase i. Organize an effective ministerial implementation e. Identify fiscal space in the health budget through h. Plan, implement and monitor to deliver on g. Plan approaches for reduced waste and f. Identify opportunities to increase efficiency in The list of indicators below demonstrates the extent of the minister s self-assessed increases in confidence. The data are presented in descending order, so that the first four demonstrate the areas in which learning and confidence increases were largest. The minister s ratings show that the largest increases were recorded in respect of using his leadership to build support for his priority goals; building a strong ministerial implementation team to plan and deliver on those priorities; and identifying opportunities to increase efficiency in budget and health resource utilization. The minister may need further assistance to understand opportunities to increase health budget effectiveness. Table 6: Changes in confidence levels by point and range increases Country Indicator Point increase Understand how to assert effective leadership in building support for my policy goals Plan, implement and monitor delivery on health system strengthening priorities Identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment Organize an effective ministerial implementation team to deliver health sector strengthening priorities Define my leadership goals to establish a transformational legacy during my term in office Identify core priorities for accomplishing my legacy Use data to monitor health delivery system performance and implementation of policy priorities Plan approaches for reduced waste and increased efficiency in procurement and supply management, and improved health worker productivity Score range 9
Indicator Identify fiscal space in the health budget through reallocation, saving and additional revenues Increase collaboration with the Finance Minister to strengthen health sector performance Point increase Hold officials accountable for performance, delivery and outcomes Understand opportunities to increase health budget effectiveness - Score range Country 6 This minister s average score increased from.6 at the baseline assessment to at the final assessment. The score range increased from between and at baseline, to for all indicators at the final assessment, suggesting that the minister felt fully equipped to undertake all actions measured in the evaluation. As shown in Figure below, she recorded learning and confidence increases for out of the indicators, and the two areas for which an increase was not recorded, had already been scored at in the baseline assessment. Figure : Comparison of individual competency scores Country 6 Competency self-assessment: Baseline and final comparison for Country 6 i. Hold officials accountable for performance, delivery and Baseline a. Define my leadership goals to establish a transformational legacy Final b. Understand how to assert effective leadership in building support for my k. Use data to monitor health delivery system performance and j. Increase collaboration with the Finance Minister to strengthen health sector c. Identify core priorities for accomplishing my legacy. d. Understand opportunities to increase health budget effectiveness. i. Organize an effective ministerial implementation team to deliver health e. Identify fiscal space in the health budget through reallocation, saving and h. Plan, implement and monitor to deliver on health system strengthening g. Plan approaches for reduced waste and increased efficiency in procurement The ratings in Table 7 demonstrate the extent of self-assessed confidence increases as recorded by the minister. The data are presented in descending order, so that the first six demonstrate the areas in which confidence increases were largest. These relate to holding officials accountable for their performance, and pursuing the health financing strategies discussed at the Forum. f. Identify opportunities to increase efficiency in budget and health
Table 7: Changes in confidence levels by point and range increases Country 6 Indicator Point increase Hold officials accountable for performance, delivery and outcomes Increase collaboration with the Finance Minister to strengthen health sector performance Understand opportunities to increase health budget effectiveness Identify fiscal space in the health budget through reallocation, saving and additional revenues Identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment Plan approaches for reduced waste and increased efficiency in procurement and supply management, and improved health worker productivity Understand how to assert effective leadership in building support for my policy goals Identify core priorities for accomplishing my legacy Organize an effective ministerial implementation team to deliver health sector strengthening priorities Use data to monitor health delivery system performance and implementation of policy priorities Define my leadership goals to establish a transformational legacy during my term in office Plan, implement and monitor delivery on health system strengthening priorities Score range In her interview after the Forum, the minister from Country 6 indicated that she does have a delivery unit in place, but that it needs to run more efficiently. She said she also needs to monitor the team more closely: Initially my plan was to monitor them every six months. I have learnt from here how to monitor them weekly. I will adapt that so that we are able to move faster. She mentioned that it is important to increase the pace of change so as to achieve her priorities prior to the elections scheduled for the end of 7. Her comments suggest that she wants to focus on improving the efficiency and effectiveness of health expenditure: The one thing that I want to tackle from here that was not strongly highlighted in my agenda was the efficiency and effectiveness of resources that we have both domestic resources and what we get internationally. I have been working on that with the international health partnerships. The processes we follow is [sic] too long. I think from what I have learnt here, I can shorten that. The minister is also keen to use evidence-based arguments to persuade her colleagues in Cabinet to view health as a sector for investment: Within our government budgeting, our lawmakers are fixed. To change their mentality is hard because every one of them wants to see their constituency in the budget. So that does not really provide a means of using the budget as efficiently as you want to use it. We do the national health accounts. But I wish I had other studies in detail like some of the ones I have seen here that would actually put out the picture on how much we are wasting and how we can use it better, and use that as a stepping stone. So I am thinking through how I can get data.
Country 7 The minister s average score across the indicators did not change as a result of participating in the Forum. The average score was. at the start and at the end of the Forum. For ten indicators the minister reported no change in confidence levels, which were high from the outset ( equipped and fully equipped ). For one indicator, notably to identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment, the minister reported a point increase in confidence (from at baseline to at the final assessment). This was also the indicator for which the minister indicated the least confidence in the baseline assessment. Notably, the minister also reported a point decrease (from in the baseline to in the final assessment) for the indicator to hold officials accountable for performance, delivery and outcomes. These results indicate that the minister felt fully equipped to take action on most of the dimensions covered by the Forum: transformative leadership, health financing (except for achieving value for money), and organizing for delivery. Since this level of confidence was present at the start of the Forum, it seems that his participation at the Forum did relatively little to change his competence assessment. The exceptions were increasing his confidence to make budget and health resource utilization more efficient, and slightly decreasing his confidence levels to hold officials to account. Figure : Comparison of individual competency scores Country 7 Competency self-assessment: Baseline and final comparison for Country 7 i. Hold officials accountable for performance, delivery and outcomes. k. Use data to monitor health delivery system performance and j. Increase collaboration with the Finance Minister to strengthen health sector Baseline a. Define my leadership goals to establish a transformational legacy Final b. Understand how to assert effective leadership in building support for my c. Identify core priorities for accomplishing my legacy. d. Understand opportunities to increase health budget effectiveness. i. Organize an effective ministerial implementation team to deliver health sector e. Identify fiscal space in the health budget through reallocation, saving and h. Plan, implement and monitor to deliver on health system strengthening g. Plan approaches for reduced waste and increased efficiency in procurement f. Identify opportunities to increase efficiency in budget and health resource
Table 7: Changes in confidence levels by point and ra8ge increases Country 7 Indicator Identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment Point increase Define my leadership goals to establish a transformational legacy during my term in office Understand how to assert effective leadership in building support for my policy goals Identifying core priorities for accomplishing my legacy Understand opportunities to increase health budget effectiveness Identify fiscal space in the health budget through reallocation, saving and additional revenues Plan approaches for reduced waste and increased efficiency in procurement and apply management, and improved health worker productivity Plan, implement and monitory to deliver on health system strengthening priorities Organize an effective ministerial implementation team to deliver health sector strengthening priorities Increase collaboration with the finance minister to strengthen health sector performance Use data to monitor health delivery system performance and implementation of policy priorities Hold officials accountable for performance, delivery and outcomes - Score range Country 8 The minister s average score increased from ( equipped ) in the baseline survey, to.9 (almost fully equipped ) in the final survey. The score range increased from between and, to between and. The minister reported increased confidence for nine indicators. The three indicators that did not indicate any increase in confidence were already rated at ( fully equipped ) at baseline and remained that way. The list of indicators below demonstrates the extent of self-assessed increases in confidence. The data are presented in descending order. The ratings indicate that minister s confidence levels increased to optimal levels across all the indicators. Table 9: Changes in confidence levels by point and range increases Country 8 Indicator Point increase Understand opportunities to increase health budget effectiveness Define my leadership goals to establish a transformational legacy during my term in office Understand how to assert effective leadership in building support for my policy goals Identify fiscal space in the health budget through reallocation, saving and additional revenues Identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment Plan, implement and monitor delivery on health system strengthening priorities Increase collaboration with the Finance Minister to strengthen health sector performance Score range
Indicator Use data to monitor health delivery system performance and implementation of policy priorities Point increase Hold officials accountable for performance, delivery and outcomes Organize an effective ministerial implementation team to deliver health sector strengthening priorities Plan approaches for reduced waste and increased efficiency in procurement and supply management, and improved health worker productivity Identify core priorities for accomplishing my legacy Score range It is very interesting that despite the minister s high ratings indicating that he felt equipped at the start of the Forum to undertake most of the actions listed, his interview refers to the very strong relevance of the Forum program to his task as health minister: I get this impression that the program was conceived by someone who appreciated my problems and challenges. It needs to be noted that increased collaboration with the finance minister was the only action for which this minister felt equipped (as opposed to fully equipped for all the others). He described the complexity this way: I now know I am not alone. Sometimes I might think am I the only one who has these problems, or my country is the only country where there are challenges reaching out to the Ministry of Finance; or the Minister of Finance appearing not to appreciate my needs. I think there is a feeling of collectiveness it is not particular to me. And that to me serves to re-energize me. He described his greatest challenge as being getting people to move because change is a painful process. The real challenge is overcoming the barriers moving people from a position of pessimism to optimism.
Figure : Comparison of individual competency scores Country 8 Competency self-assessment: Baseline and final comparison for Country 8 i. Hold officials accountable for performance, delivery and outcomes. Baseline a. Define my leadership goals to establish a transformational legacy Final b. Understand how to assert effective leadership in building support for my k. Use data to monitor health delivery system performance and j. Increase collaboration with the Finance Minister to strengthen health sector c. Identify core priorities for accomplishing my legacy. d. Understand opportunities to increase health budget effectiveness. i. Organize an effective ministerial implementation team to deliver health e. Identify fiscal space in the health budget through reallocation, saving and h. Plan, implement and monitor to deliver on health system strengthening g. Plan approaches for reduced waste and increased efficiency in procurement f. Identify opportunities to increase efficiency in budget and health resource Country The minister s average score across all action indicators increased from. at the baseline assessment to at the final assessment, suggesting that he felt fully equipped to undertake action in the areas measured in the competency profile. The score range increased from between and at baseline, to at the final assessment. The minister s confidence increased in out of areas; the remaining two areas had already been rated ( fully equipped ) in the baseline assessment and remained thus in the final assessment. The largest increase in confidence was for the indicator plan approaches for reduced waste and increased efficiency in procurement and supply management, and improved health worker productivity, which moved from a baseline score of ( not at all equipped ) to an end-line score of ( fully equipped ).
Figure 6: Comparison of individual competency scores Country Competency self-assessment: Baseline and final comparison for Country Baseline Final i. Hold officials accountable for performance, delivery and outcomes. k. Use data to monitor health delivery system performance and j. Increase collaboration with the Finance Minister to strengthen health sector a. Define my leadership goals to establish a transformational legacy b. Understand how to assert effective leadership in building support for my c. Identify core priorities for accomplishing my legacy. d. Understand opportunities to increase health budget effectiveness. i. Organize an effective ministerial implementation team to deliver health sector e. Identify fiscal space in the health budget through reallocation, saving and h. Plan, implement and monitor to deliver on health system strengthening g. Plan approaches for reduced waste and increased efficiency in procurement f. Identify opportunities to increase efficiency in budget and health resource The list of indicators below demonstrates the extent of self-assessed increases in confidence. The data are presented in descending order, so that the indicators demonstrate the ten areas in which confidence increases were largest. These were in the areas of health financing, increasing efficiency of health spending, and building support for his priority goals. Table : Changes in confidence levels by point and range increases Country Indicator Plan approaches for reduced waste and increased efficiency in procurement and supply management, and improved health worker productivity Point increase Understand how to assert effective leadership in building support for my policy goals Identify fiscal space in the health budget through reallocation, saving and additional revenues Identify opportunities to increase efficiency in budget and health resource utilization to achieve better value from health sector investment Plan, implement and monitor delivery on health system strengthening priorities Organize an effective ministerial implementation team to deliver health sector strengthening priorities Increase collaboration with the Finance Minister to strengthen health sector performance Score range 6
Indicator Use data to monitor health delivery system performance and implementation of policy priorities Point increase Understand opportunities to increase health budget effectiveness Hold officials accountable for performance, delivery and outcomes Define my leadership goals to establish a transformational legacy during my term in office Identify core priorities for accomplishing my legacy Score range In his interview the minister commented on the relevance of the Forum and described its impact as follows: I have been re-energized to go back home and face up to the tasks of being a deputy minister in such a complex sector and government structure. Now I know how to focus. I know how to go back and reorganize my thoughts around priorities and various action steps that need to be taken to reshape the health sector in our country. I have been sharpened to be a better leader of the ministry. What is likely to change? Five of the health ministers were interviewed at the end of the Forum to gain further insight into the impact on the development of their ministerial competencies. The ministers shared specific ways in which they felt better equipped for their role in government, what they would do differently, how they would deal with challenges, and prospects for cooperation with their finance minister. One minister summed up the impact of the Forum in this way: Now I feel I have more confidence in tackling the task of leading such a complex ministry within government. (Country ). Impact on the ministers leadership role Some ministers commented that the Forum had effectively combined big picture thinking with practical ways in which to achieve their legacy goals: to be able to set the big picture agenda and how to achieve it (Country 6). Defining a legacy was a major step for all the ministers who were interviewed. One put it this way: I learnt how to plan activities according to our legacy. Before coming here, I was not thinking about legacy. I had a lot of things that I wanted to do because in our country we have a lot of priorities. From this forum, I have learnt that among the priorities we have, I should focus on the one thing that people will remember me by. This has been very interesting to me. I have started to think how I can improve the management of my ministry. Now it is clear to me that my legacy should be [the] total eradication of maternal and child mortality. (Country ) Another minister also referred to the value of realizing a vision by prioritizing and building a support base: Initially I thought whoever does not agree with my vision is my enemy. I have learnt that it is not necessarily the case and you can use your perceived enemies, bring them on board and they can add more value. (Country 9) 7
He also described the challenge of making the transition from being a health professional to becoming health minister: I was a doctor in a hospital and all of a sudden I became a minister. It is a big transition from being a professional to now having to manage people. So I find this course very useful. As a minister I have priorities, but I did not have the fine details on how to navigate the roadmap to arrive at those priorities. I was not equipped with tools. This course has helped me. (Country 9) The importance of transparency in successful leadership was mentioned as an important lesson learnt: Another aspect is to be transparent, not keeping the information inside, but putting it out there for the population that you are serving, to know when you are making headway and when you are not. That way, they will stop asking because they follow what you are doing. (Country 6) The overall relevance of the Forum to the challenges of ministerial leadership was described very clearly by one minister: What is particularly interesting to me is I get this impression that the program was conceived by someone who appreciated my problems and challenges. And that gives you a real life situation. For example, how do I define my legacy? How do I break barriers? How do I move the entire ministry to align with that agenda? So it is very practical to me and I think that is the most interesting thing. Each day I feel they appreciate my problem. (Country 8) These perspectives are consistent with the Net Confidence Ratings recorded by the whole group for the leadership components of the Program before and after the Forum (as shown alongside and also in more detail in Figure, section.). Across all three indicators, the 9 ministers who completed the final competency assessment at the end of the Forum moved from feeling somewhat equipped and equipped to feeling fully equipped to carry out different leadership roles in strengthening their health systems. 8