Appendix C: Econometric Analyses of IFC and World Bank SME Lending Projects: Drivers of Successful Development Outcomes IFC Investments RESEARCH QUESTIONS Do project characteristics matter in the development outcome? Does the product line of the intervention have an impact on the development outcome? DATA To address these two questions, IFC project-level data were used. Documents were pooled and projects rated across a number of indicators. In the end, a total of 103 SME targeted projects were rated and subject to analysis. VARIABLES CONSTRUCTION AND ESTIMATED STRATEGY To address these issues discussed at the beginning, we mainly consider the variation of projects development outcome (DO) as a function of two types of variables: Country-level condition under which SME projects are implemented, including income level Project-level conditions, which involves supervision, risk management, monitoring and supervision of loans, as well as duration, sector, product line and loan size (Kilby 2000). Accordingly, the following model specification is to be estimated. Development outcome = a + bx + kw + hp + iz + e, where X is the country-level variable, w is the control for project characteristics such as sector, length of project, and size of project, P is the product line/intervention type, and Z is a vector of the project-level variables. The e is the random error term, normally distributed. 179
The DO is captured by an IEG rating scheme, which has value from 1 to 6, with 6 being highly satisfactory. As noted above, the main predictors in the model are the project characteristics and project-specific characteristics. Aggregate country level variable, X, is captured by the country income level variable, presented as a series of dummies. The product line/intervention, P, is also captured by a series of dummy variables: funds, investment in SME, leasing, onlending, and others, with others as the omitted group in the model. There are four main project characteristics, Z, brought into analysis: SUP supervision, a constructed variable Inadequate technical design, which is presented as binary variable (yes or no) Inadequate risk management, which is also presented as a binary variable Number of problems observed by IEG in the project. The means and standard deviation (plus and min and max value) of each of these variables are shown in the Table C.1. Two ordinary least squares regression models are presented. First, the IEG DO variable is regressed on X (country-level income variable), W (basic project information such as length and size of project and industry composition), and P (product line/intervention). In the second model, other project characteristics are included. This two-step estimation strategy with increment of R-sq will give some insight as the role of project characteristics on the IEG rating. Findings Descriptive statistics (Table C.1) show that among the IFC TSME sample of 103 projects, the majority of them were in the financial management industry, accounting for 89 percent. Projects from manufacturing, agriculture, and services industry and infrastructure industry account for 7 percent and 4 percent, respectively. The duration of projects averaged 4.5 years. Most projects were in the lower-middle-income, upper-middle-income, and highincome countries, accounting for 34, 38, and 20 percent, respectively. Only 8 percent of projects were in low-income counties. 180
By product line, on-lending accounts for 59 percent of projects. The second most popular product line was funds, which accounted for 16 percent. In terms of project characteristics, the IEG rating scheme suggests that 26 percent of projects were identified as having an inadequate design; 23 percent were identified as having inadequate risk management. (Note: The whole list of problems identified by IEG included inadequate risk assessment, inadequate technical design, inadequate supervision, inadequate political or institutional analysis, inadequate baseline data or unrealistic targets, inadequate M&E framework, poor data quality, inadequate partner financing or coordination, implementation disrupted by a crisis, and project restructuring.) The average score for supervision is 2.85, which is close to satisfactory (3 in the rating system). The average rating for DO is 3.7, which is between moderately unsatisfactory and moderately satisfactory. The main findings of the analysis are presented in Table C.2. In the baseline model three findings emerge: Projects in upper-middle-income countries generally have higher DO ratings than projects in high-income countries; however, projects in low-income and lower-middle-income countries are not significantly different from highincome countries, other things equal. Length of project from initiation to maturity seems to be positively related to the DO rating. Among product lines, investment in SME seems to have significantly positive association with IEG rating at the 10 percent levels; however, this difference becomes insignificant in model 2, which controls for project-relevant characteristics. Model 2 (column 2) includes project-relevant characteristics. One obvious finding is the significant increase of R-square value from 0.271 (model 1) to 0.468 (model 2). This clearly suggests that relevant project characteristics are important predictors of the IEG rating. This finding is consistent with recent findings that s striking feature of the data is that the success of individual development projects varies much more within countries than it does between countries (Denizer, Kaufmann and Kraay 2011). 181
More specifically, model 2 suggests that project supervision quality is positively correlated with IEG s DO rating. One unit increase in the rating of supervision is associated with a half of unit increase in IEG s rating. In addition, a project that has a problem of inadequate risk assessment is associated with one unit lower IEG DO rating as compared to a project without the problem. A project identified as having an inadequate design is associated with a two-thirds of a point lower IEG DO rating. Model three (in column 3) is an extension of model 2, where two specific project problem variables are replaced by one variable, the number of problems observed. The results further suggest that one problem observed is associated with one-third unit lower IEG DO rating. Finally, the results indicate that controlling for these additional factors, on-lending is associated with significantly better development outcomes than other product lines. Table C.1. Means and Standard Deviations of Variables in the Analysis Variables mean std min max (n=103) IEG_rating 3.70 1.22 1 6 Duration 4.53 0.65 3 7 Loan size (in log) 9.20 1.80 0 12.52 Product line/intervention Funds 0.16 0.36 0 1 Investment in SME 0.08 0.27 0 1 Leasing 0.10 0.30 0 1 On-lending 0.59 0.49 0 1 Other 0.07 0.27 0 1 Sector FM industry 0.89 0.31 0 1 Infra industry 0.04 0.19 0 1 MAS industry 0.07 0.25 0 1 Country income level Low income 0.08 0.27 0 1 Lower middle 0.34 0.48 0 1 Upper middle 0.38 0.49 0 1 High income 0.20 0.40 0 1 Project characteristics SUP Supervision 2.85 0.56 2 4 Inadequate design (yes) 0.26 0.44 0 1 Inadequate risk mgmnt (yes) 0.23 0.42 0 1 Number of problems flagged 1.88 1.46 0 6 Source: IFC projects data. 182
Table C.2. OLS Regression of IEG Outcome Rating: Role of Project s Characteristics Model Specifications Predictor 1 2 3 Constant -0.613 0.020-0.321 Length of project in year 0.499*** 0.408** 0.420** Loan size in USD (in log) 0.028 0.002 0.072 Product line/intervention a Funds 0.142 0.311 0.028 Investment in SME 1.105* 0.133 0.232 Leasing 0.433 0.361 0.618 On-lending 0.799 0.672 0.917* Sector b FM industry 1.098* 0.219 0.118 Infra industry 0.175 0.301-0.132 Country income level c Low income 0.183 0.432 0.172 Lower middle 0.156 0.162 0.128 Upper middle 0.617** 0.584** 0.653** Project characteristics SUP Supervision --- 0.497*** 0.384* Inadequate technical design (yes) --- -0.679*** --- Inadequate risk assessment (yes) --- -1.005*** --- Number of problems flagged out --- --- -0.318*** R-sq 0.271 0.468 0.422 N 94 94 94 Source: IFC projects data. Note: Dependent variable is the IEG rating, which ranges from 1 to 6, with 6 being the highly satisfactory. Supervision ranges from 1 to 4, with 4 as excellent. a The omitted group is other category. b The omitted group is MAS industry. c The omitted group is high income countries. ***p<0.01; **p<0.05; * p<0.10. 183
World Bank Investments The main questions to be addressed are threefold: In seeking to understand the project-relevant factors that are associated with successful development outcomes, the team sought to answer three questions: How does the type and characteristics of TSME intervention relate to DO ratings? How does country income level relate to DO ratings? How do measured design, oversight, and evaluative variables relate to DO ratings? EMPIRICAL IMPLEMENTATION To address these three issues, the variation of projects development outcome as a function of country income where the project was implemented and project-specific characteristics. In terms of country conditions, a strong institutional setting could ensure a better and efficient implementation, leading to better development outcomes (Khwaja 2009; Kraay 2010; Rajan and Subramanian 2008). Project-specific characteristics include the type of project (product line), length (in years), and size (in dollars), and variables related to design, implementation (and/or supervision) and monitoring and evaluation (Kilby 2000). In light of three questions raised in the evaluation, the following model specification is to be estimated. Development outcome = a + bx + iz +kw + e, where x is the country-level variable, Z is a victor of the project-level variables, and w is the controls for particular loan characteristics. The e is the random error term, normally distributed. The DO is measured by IEG outcome rating, ranging from 1 (highly unsatisfactory) to 6 (highly satisfactory). Ordinary least squares regression will be used for estimation. 71 The main predictors in the model are the intervention type, which is captured by a series of dummies, and whether loans are business development services/technical assistance, matching grants, line of credit, or other. Another set of predictors is project characteristics, captured by four variables: (i) number of risks for a project s monitoring and evaluation (M&E); (ii) number of risk 184
flags for project management; (iii) number of risk flags for slowness of disbursement; and (iv) whether the project was identified as having an overly complex design. 72 Country income level is represented in four categories: low, lower middle, upper middle, and high. An alternative World Bank classification variable was tried and dropped. MAIN FINDINGS Descriptive statistics are presented in Tables C.3 C.5. Table C.3 contains the analytical findings, with two model specification to answer the three policy questions. Model 1 of Table C.4 The analysis does not show a significant association between intervention type and IEG outcome rating. TSME interventions in lower-middle-income countries are less effective in IEG outcome rating than loans that go to upper-middle-income countries, other things being equal. The loans to SMEs in low-income countries are not significantly different in outcome. 73 Project characteristics that have significant association with outcomes are (i) overly complex design, (ii) flags for the way in which the project was managed, and (iii) flags for slow disbursement. The flag for weak M&E had a negative relation to IEG rating, but its coefficient is not significant. The role of slow disbursement in the project is positive, which is quite surprising. It seems to indicate that taking time in disbursement can be associated with better outcomes. This merits further exploration, as other explanatory variables may be important and omitted. Model 2 In model 2, an additional two variables are controlled for: length of project in years and log of size of project in millions of dollars in log to see if the results change. The length of project is constructed by difference in years between the project approved and project completion. The mean of length is a little over four years and the average size of loans is $178 million. Controlling for these two variables does not change the main findings from model 1, although both length and loan size are negatively associated with the IEG rating. 185
Table C.3. Frequency Distribution of Intervention Type Intervention type n % BDS/TA 24 51.06 LoC (line of credit) 12 25.53 Matching grants 7 14.89 Other 4 8.51 Total 47 100 Source: World Bank lending project. Table C.4. Mean of Variables Variables mean std (n=47) Overly complex design 4.23 0.89 Flag M_E 0.17 0.48 Flag project management 0.26 0.61 Flag slow disbursement 0.60 1.35 Length of project in year 4.26 3.05 Size of loan in ($ millions) 178 214 Source: World Bank lending project. Table C.5. Means of IEG Development Outcome Score by Intervention Type Variables n mean std Intervention type BDS/TA 24 4.29 0.91 LoC (line of credit) 12 4.17 0.83 Matching grants 7 4.00 1.15 Other 4 4.50 0.58 IDA classification IDA/blend 29 4.28 0.92 Non-IDA 18 4.17 0.86 Country income grouping High income: non-oecd 2 4.50 0.71 Low income 10 4.40 0.84 Lower middle income 18 4.00 0.69 186
Upper middle income 17 4.35 1.11 Total 47 4.23 0.89 Source: World Bank lending project. Note: The IEG development outcome coding ranging from 1 (highly unsatisfactory HU ) to 6 (highly satisfactory HS ). 187