Version: 28 June 2016 For Information CGIAR Consortium CRP2 Value for Money (V4M) Analysis Purpose: This paper provides, as a companion document to the Consortium Office prepared paper titled Developing the Rationale for Investing in the 2017-2022 CRP2 Portfolio (document CB26-08, a draft set of materials for broad stakeholder review and input), an explanation of the value for money analysis carried out by the Consortium, include the methodology used and the functionality of the online tool used for the submission process. It considers its role in contributing to the overall analysis of the submissions of full proposals as part of the 2017-2022 portfolio of CGIAR Research Programs. This document can be shared by CB participants with Centers for input in advance of Board deliberations. Document Category Standard Document Montpellier, France, 20-21 June 2016 Page 1 of 8
What Value for Money is and is not, in the context of the CRP2 Portfolio: In this document, we start by describing what this analysis is composed of, then lay out the methodologies and visualization tools, and lastly present the data collection effort to support the analysis. The term value-for-money can mean different things to different people. We lay out in this document what value-for-money means in the context of the CRP2 portfolio analysis. Value for Money, in the context of the CGIAR CRP2 portfolio, is an attempt to link the results (preferably outcomes or milestones towards targets) to the investments required to produce such results. No attempt is made to standardize a methodology to calculate returns on investment using standard economics tools such as Internal Rate of Return, Net Present Value or Benefit Cost Ratios. Instead, the focus is on standardizing a set of results (outcomes) against two evaluation frameworks defined in CGIAR s 2016 2030 Strategy and Results Framework ( SRF ): respectively, the ten CGIAR targets, and the Results Framework (a quantitative and qualitative evaluation framework). The time frame is constant for all proposals (6 years; 2017-2022) and no efforts have been made to discount either investments or results produced at different points during this period. Rather, all investments are assumed to be more or less evenly spread over the 6 years, and the results are assumed to be produced at the end of the period, in 2022, as milestones of progress towards achieving 2030 targets, where relevant. All CRP proposals have been asked to specify results in terms of standardized targets, and to specify investments against these targets and against the IDOs and sub-idos in the Results Framework. Results and investments are specified at component / flagship level, and aggregated at program level. It is clear that they are interdependent, i.e. some activities produce multiple results (e.g. increased productivity as well as reduced poverty), and some activities are linked, and are jointly required to produce a single result. While there are clear limitations to the approach, as a minimum it enables the following types of analysis and inquiry: 1. Transparently and explicitly linking results to activities and investments, thereby facilitating ISPC to query and validate the assumptions underpinning either the results and/or the proposed budgets. 2. The same data support an analysis on the coherence of budgets and results across flagships and programs. It also supports an analysis of the portfolio interactions between CRPs, within geographies and with the platforms. 3. The same data also supports an analysis of the projected results (outcomes, eventually leading to impacts) based on proposed investment at the level of the portfolio. Thus, it addresses the important question of whether in principle, the Montpellier, France, 20-21 June 2016 Page 2 of 8
targets set in the SRF can be achieved and thereby giving credibility to the proposals and the overall portfolio. 4. The data analysis also supports prioritization of investments. Visualizing proposed investments against targets, against IDOs and sub-idos of the results framework, and by geographies (both site integration and other countries) allows a more transparent interpretation of proposed or planned investments, bringing more rigor and confidence to the choices made by investors. The CRP2 Portfolio [presented by the Centers] is a complex set of 12 programs and 3 platforms, about 70 flagships and modules, 20 site integration geographies, more than 40 participating partners and numerous collaborations. The value-for-money analysis provides a crisp and synthetic overview of how all these component pieces are combined to make a powerful portfolio of investments with ambitious goals. Ultimately the approach outlined in this document provides a strategic framework to assist donors with their investment decisions for CRP2. Description of the value-for-money package The value-for-money analysis combines the CRPs aspirational results and anticipated needs for financial resources over the next six years into a set of visuals. These visuals provide strategic insight into the overall portfolio s expected results, investment needs, geographic coverage, and role clarity and coordination amongst the different players. The analysis produced two main outputs: 1. The detailed value-for-money data package provided to the ISPC provides reviewers with tools to drill up and down between flagships, outcomes, and sub-idos investments, to assess the coherence and consistency between the description of the research, the cluster of activities, the flagships and the overall CRP aspirational results. The CO provided this material to assist ISPC in its review. 2. The aggregated value for money overview at portfolio level that is contained in Chapter 5 of the Consortium s Investment Case for the CRP2 Portfolio document, and shared with other interested stakeholders ultimately a tool for resource mobilization (as a more comprehensive source for derived, target specific, shorter products to be used for resource mobilization efforts). Montpellier, France, 20-21 June 2016 Page 3 of 8
Methodology and visualization techniques The V4M analysis is organized against two results frameworks, both adopted as part of the Strategy and Results Framework (SRF) in 2015: a. CGIAR 2022 intermediary targets These targets have been written in the context of the SDGs with 2030 quantitative goals, as well as 2022 intermediary goals. b. Results Framework (IDOs and sub-idos) This framework was designed to help describe and categorize the types of results that each flagship projects aims to accomplish. The analysis presents results and investments against these frameworks to visualize how the various parts of the CRPs combine to make a coherent and synergistic portfolio. For this to be possible, a number of steps were undertaken: 1) We first needed to harmonize as much as reasonably possible how the CRPs would describe the combined results and investments of their programs. We defined guidelines and provided standard forms and tables for the CRPs to submit with their proposals in the Guidance document. There was still a fair amount of gray areas particularly around harmonizing the methodologies for quantifying contributions to the targets. An area of future improvement is to further increase the alignment between the CRPs on these methodologies. For example, some CRPs counted the entire investment amount of their program against the CGIAR quantitative targets. Others decided to only count those investments that directly contributed to the targets, and exclude some or all of the supporting investments (for example crosscutting investments in capacity building). Despite these differences, still over 80% of the CRP2 total investment is counted towards accomplishing the CGIAR targets. 2) We then needed to design powerful visuals that would distill the large amount of quantitative information into single synthetic views. Montpellier, France, 20-21 June 2016 Page 4 of 8
Here are some examples of the visualization techniques we used: Thumbnail Description Heatmaps: To visualize the funding intensity of each CRPs to the targets, IDOs or sub-idos. IDO or sub-idos levels of investment: To visualize the top and bottom IDOs or sub-idos in terms of investments received. Synergies: To visualize the handshakes for collaborations between CRPs by targets. Portfolio s total contribution to targets: To estimate the total contribution from the individual CRPs contribution by correcting for double counting. Contribution to target vs Investment quadrant: To identify the positioning of individual CRPs by target. Geographic footprint of the portfolio: To visualize on a map where the CRPs are integrated together by target. Top geographies: To visualize the top geographies in terms of impact while distinguishing between site integration and other countries. Financial analysis: To visualize the budgetary information such as cost structure, funding gap, mapping between sources and uses of funds, etc. Montpellier, France, 20-21 June 2016 Page 5 of 8
Avoiding double counting: A Conservative approach Note that to add up each CRP s contribution to an overall portfolio contribution to each of the targets, we made a simple assumption to avoid double counting the impact of multiple CRPs. We assumed that when multiple CRPs contribute to a given target in the same country, they all reach the same farmers, the same people, the same land or forest area and therefore we would only keep the maximum of all these CRPs contribution. For example, if Maize and Wheat aspire to reach 10 and 5 million farmers respectively in India for adopting new varieties, then we assumed that together they would reach 10 million farmers, not 15 million. Avoiding this double counting is a complex problem and our approach is simplistic and conservative. We believe however that it gives us an order of magnitude to get a picture of the overall portfolio that can be refined over time. Supporting analysis: Detailed financial review The value-for-money analysis is underpinned by a detailed financial review of the flagships budgets. The financial analysis provides information on the total financial resources needed, the timing of the funding needed, the funding gap and needs for fund raising of W3 and Bilateral. It also provides information on the uses of funds towards internal resources (such as personnel, travel, etc.) and external collaborations, and the total management and support cost of the overall program. Last, it provides information on how individual flagships have prioritized the uses of W1+2 funding. This analysis will be used to provide feedback to the CRPs on the final budgets to be submitted for approval. Data collection The table below describes the data and level of detail that had to be submitted by the CRPs as part of their proposal, for each results framework: Level of program hierarchy CRP Flagship CGIAR 2022 Intermediary targets Overall Contribution to targets Contribution to targets by countries Investment amount by target, differentiated by windows of funding Synergies between CRPs by target SRF Investment amount by sub- IDOs, differentiated by windows of funding (Base budget only) Description of outcomes (Base budget and uplift budget) Montpellier, France, 20-21 June 2016 Page 6 of 8
In addition, CRPs were provided one standard budget template in excel and asked to submit a detailed budget by participating partners and flagship projects, including a list of expenses by natural categories (personnel, travel, collaborations, etc.) Online Submission Tool To collect this data, an online submission tool was developed to ensure standardization of the proposal submissions from all CRPs and Platforms, and facilitate the value-for-money (V4M) analysis across the portfolio. While the online tool had some technical issues with data entry and printing for the narrative sections, the databases exported from the tool for the V4M analysis, actually proved to be very useful. It produced an aggregated database of the CRPs and Platforms Budget roll-ups, detailed flagship budgets, contribution to the targets by geographies, contributions to the SRF and detailed flagship outcomes. For the budget roll-up, the online tool read and consolidated almost 70 Flagship / Module budget excel files. This database was key for the Budget analysis across the various windows of funding. The database of the PIM tables was also key for the SRF and Intermediary targets analysis. Following submission, the databases from the online tool were connected to Tableau, a powerful data visualization tool. This tool provided an initial visual overview of the data, which made us quickly realize that an extra step of data validation was required to ensure data accuracy of the budgets and/ or PIM tables submitted and consistency across databases. In fact, this data validation exercise proved to be a significant effort in the end. To highlight some examples, when analyzing the data, we noticed that: some CRP s submitted W1+W2 funding that was higher or lower than originally assigned some CRPs included the Management and Support costs in the outcomes (PIM B) analysis and sub-ido (PIM C) tables, which was not required. In some cases, too, the total amounts requested in these tables were not in-line with their budgets. some CRP s submitted numbers in a different format, which we had to fix manually to be able to analyze the data accurately across all CRP s, i.e. 1,000,000 to 1, and some CRP s had a different understanding of what we required in the Contribution to 2022 CGIAR Targets (PIM A) table. As part of this data validation phase, we communicated with the individual CRPs to highlight inconsistencies, and clarify misconceptions or misunderstandings of what was required. Seven CRP s ultimately re-submitted their budgets and / or PIM tables through the online tool in April 2016. The final consolidated databases were then shared with the ISPC on 24 April 2016, to ensure the latest information submitted was available to be considered in their proposal reviews. Montpellier, France, 20-21 June 2016 Page 7 of 8
Conclusions The value-for-money analysis was initially designed to answer four key questions, ultimately informing the donors investment decisions for the CRP2 Portfolio: 1) What share of the CGIAR 2022 intermediary targets is the portfolio positioned to deliver? 2) Are there gaps in the portfolio? What are the complementarities or redundancies? 3) How is W1/W2 proposed to be used and prioritized? How much are the outcomes reliant on different sources of funding? How much needs to be fundraised/secured? 4) What are the major risks to the CRP2 portfolio s success? Which risks are we willing to take? For some risks do we need mitigation strategies? We think that the value-for-money analysis and tools successfully answer questions 1 and 2. For question 3, as the Centers and CRPs decided in their Rome meeting in November 2015 not to prioritize W1-2, but to allocate an equal amount of W1 to each CRP and CRPs appear to use the W1-2 by and large as spread out base funding, rather than to prioritize specific this question does not have a useful answer yet. The donors have indicated that they do want to see a prioritization of W1-2 at FC15 but that is homework still to be carried out by the Centers/CRPs. To answer Question 4, we need to create a risk framework or register that can be operationalized and tracked on a regular basis, that identifies key learnings, progress and failures and allows on-going reprioritization of resources to avoid throwing good money after bad, while preserving a strong spirit of calculated risk taking to support innovation. Refinement of the targets would be a distinct plus, e.g. the need to provide baselines for target 6; but also the translation of the targets into lower level indicators (as per the MEL Task Force or CoP) with which we might monitor progress. Such refinement should precede the risk analysis. We hope that this analysis will help donors and reviewers draw out new insights and conclusions when considering the CRP2 portfolio and help strategically position the proposals and clarify the roles of the individual players. The challenge and opportunity for the value-for-money analysis and tools is to remain relevant and useful into the future. To achieve this, the current V4M analysis needs to be converted into robust project management tools that are owned by the CRP leaders, practitioners and investors to enhance the overall success of individual CRPs and the portfolio. Montpellier, France, 20-21 June 2016 Page 8 of 8