Evaluating the impact of agricultural credit: A matching approach

Transcription

1 Evaluating the impact of agricultural credit: A matching approach Sunil Mitra Kumar August 15, 2012 Abstract Agricultural credit forms an important plank of financial inclusion policies in developing countries, yet evaluations of its impact on borrowers wealth status are very few owing to significant problems of identification. Using national data from India that constitutes a short panel, I use matching techniques to evaluate this impact for a sample of farmer households, by matching on covariates that were measured before a proportion of these households availed of loans. I then select the most suitable matching technique based on the extent of covariate balance that results from matching, as evaluated by standardized bias plots, quantile-quantile plots, and summary measures of the latter. I find evidence of a positive treatment effect and discuss the robustness of the findings through checks for sensitivity and consistency. Key Words: Matching, Covariate balance, Agricultural Credit, India School of Economics, University of East Anglia, Norwich, U.K. NR4 7TJ; Sunil.Kumar@uea.ac.uk

2 1 Introduction Providing farmers with access to credit has long been a focus of policy in developing countries because in the absence of well-functioning markets for credit, investments that might otherwise yield high rates of return often remain untapped. In agriculture, the mainstay in several developing economies, such investments include fertilizers, pesticides, better seeds and irrigation technologies. Rural India, which is the focus of this paper, has seen several decades of government policy focus on expanding the network of banks, and urge banks to provide farmers with credit. However research on the impact of such credit on poverty and income has remained sparse, which is surprising given the importance placed on enabling access to credit, but explained perhaps by the significant problems of identification any such widely and diversely implemented policy presents. There are two major exceptions to general paucity of research on the impact of credit. The first is a study by Burgess et al. (2005) who exploit spatial and chronological variation in a bank branch-expansion program that in India from 1977 to 1990, and find that the expansion of banks in rural areas significantly reduced poverty. The second is a significant and growing number of studies that evaluate the impact of micro-finance. As a relatively new phenomenon, the expansion of micro-finance lending is amenable to randomized and quasi-experimental allocations of credit, spatially or across time, features that Pitt and Khandker (1998) and Banerjee et al. (2009) have exploited. Large-scale and often public-sector programmes with non-random treatment allocation are difficult to evaluate using conventional econometric techniques, since the decision of whether or not to apply or participate in the program, given a set of eligibility criteria, confounds attempts to evaluate impact. In the absence of a credible exclusion restriction that enables identification of the participation decision separately from the program s impact, any estimate of the effect of the program might be biased. Thus while an instrumental variable correlated with program participation and independent of the outcome (other than via program participation) would provide the required exogenous

3 variation, such variables may not exist, or might be difficult to measure in a survey. Evaluating the impact of agricultural credit in the Indian context throws up similar challenges, which perhaps explains the paucity of research on evaluating treatment effects. Borrower-level eligibility criteria do exist in theory, such as the requirement that farmers should own land that can be used as collateral, but there are situations in which these are not adhered to, as indeed the data used in this study confirm. This can be due to advisory guidelines from the Reserve Bank of India (RBI) that instruct banks to relax the requirements for collateral or security 1, or sporadic government directives aimed at expanding access to credit for political reasons (Cole, 2009). In addition, since the end of the bank expansion program in 1990, there has been no comparable policy-emphasis on expanding the number bank branches, making it difficult to use variation in the density and expansion of branches to estimate the impact of credit. Finally, while randomized allocations of credit could enable estimation of treatment effects, implementing this for loans from what are mainly government banks would be unfeasible, being both unethical and also impractical given the politically sensitive nature of bank credit in India. Matching estimators can overcome some of these methodological challenges, provided suitable data are available. Conditional on the assumption that treatment status is random once observables have been accounted for, matching enables the measurement of treatment effects for a subset of the sample that for which suitable matches can be found. Matching techniques have long been used in the medical sciences, and more recently, in a growing number applications in the social sciences, such as measuring the impact of piped water on reductions in diarrhea (Jalan and Ravallion, 2003), of microfinance loans in mitigating poverty (Imai et al., 2010), and of migration on wages (Ham et al., 2003). Overwhelmingly though, matching techniques have been 1 As a circular issued by the Reserve Bank of India in 2004 suggests (RBI 2004), and as part of priority sector lending for Scheduled Caste/Scheduled Tribe borrowers who have less than 5 acres of land or are landless labourers (RBI 2002) 2

4 applied to evaluate large-scale and often public-sponsored job training programs, such as Dehejia and Wahba (2002) who compared matching to randomized controlled trials 2, Heckman et al. (1997), Sianesi (2004) and Lechner et al. (2011). Using data that are nationally representative and form a short panel, I use matching estimators to measure the impact of formal agricultural bank credit on farmers wealth status. Each household was visited twice over the course of the survey, and the data provide an effective before-after period of one year for certain key variables. I restrict the analysis to households who did not have a loan on the first visit, and define treatment and control status according to whether they obtained a loan between the first and second survey visit. I then match households according to the propensity score and Mahalanobis distance metrics, using covariates that were measured before the treatment (obtaining a loan) and on covariates that are independent of treatment status, such as demographic variables. I then select the most appropriate matching method(s) based on the amount of balance that results after matching. In the social sciences, balance is usually evaluated through a set of t-tests, though this practice has been criticized and shown to be inappropriate since the resulting test statistics are sensitive to sample size, and are based on the assumption that the matched samples are in fact drawn from a larger population (Imai et al., 2008). I therefore use standardized bias plots, quantile-quantile plots and summary measures of the latter to evaluate balance, select the most appropriate matching techniques accordingly, and find evidence of a positive treatment effect. Finally, I evaluate the sensitivity of this treatment effect to different assumptions, including that of strongly ignorable treatment assignment. 2 See also the resulting debate on whether this comparison was successful between Smith and Todd (2005) and Dehejia (2005). 3

5 1.1 Agriculture and credit Agriculture in India takes place in two main seasons. The earlier Kharif season is dependent on the monsoon, and runs from approximately June to September. The second is the Rabi seasons starts at the finish of Kharif, and runs through the winter, and harvests usually take place in spring. Despite a continuously declining contribution to national income, agriculture remains the single largest employer in the Indian economy. Credit is widely understood as a vital agricultural input since it permits the farmer to purchase tangibles such as seeds, irrigation, fertilizer and pesticides, credit. Since the agricultural production cycle lasts for a few months and consequently investments are required far before income can be realized, credit is usually required to purchase various inputs. In the absence of sufficient personal financial resources a farmer must look towards formal or informal sources of credit, with the clear implication that the terms on which such is obtained have a direct bearing on the profitability of agricultural production. Informal sources of credit are generally considered undesirable in the Indian context given that interest rates tend to be unreasonably high, and as a result, the proportion of net agricultural credit that was supplied by banks (as opposed to informal sources) is taken as an important indicator of the success of agricultural credit policy 3. Over the last half century, the Indian government and Reserve Bank of India have taken an active interest in agricultural credit, and successive policies have focused on expanding the provision of agricultural credit through targeted lending at subsidized rates, compulsory sectoral lending targets for banks, and importantly, expanding the network of bank branches. 14 major commercial banks were nationalized in 1969, enabling the government to pursue social objectives Pande (2007), and subsequently, when it was felt that nationalization alone was not sufficient to achieve these aims, a social banking program that involved a large expansion of bank branches in previously 3 See, for instance, the Report of the Committee on Financial Inclusion (NABARD, 2008) 4

6 unbanked rural locations was undertaken over the period from 1969 to 1990 that forced banks to open a number of rural branches for each new urban branch they were allowed to establish (Burgess and Pande, 2005; Burgess et al., 2005). In terms of lending, the Reserve Bank of India has long made recommendations for priority sector lending, a sector of which agricultural credit forms the central part. These included a 40% target for priority sector lending from the overall lending portfolio (in 1985) which has since been reduced, even though banks continue to be encouraged to lending to the priority sector (RBI 2008). Whether credit impacts rural incomes is thus a question of paramount importance, and we might expect suitably priced credit from formal providers such as banks to enable farmers to purchase agricultural inputs, repay loans after harvest, and overall benefit from access to bank credit through an increase in the income generated from agriculture. An increase in income on account of access to bank credit might or might not be revealed by different indicators of income and wealth. A farmer might simply save the extra income as cash, and use this for investment in subsequent agricultural production cycles, or extra income might lead to an increase in consumption expenditure on a variety of goods and services. Equally, extra income might be used to purchase livestock, machinery or land as productive investments for subsequent agricultural production. I use an index of consumer durables as the indicator of wealth or income status, as discussed in detail in section 3, based both on the existing literature on measuring wealth in developing-country contexts, as also the need to enable comparisons across the largest possible sample by using variables that are common to all farmers. 2 Estimating the impact of loans Our aim is to estimate the impact of bank loans on an indicator of wealth or income. A standard representation of the impact evaluation problem 4 is as 4 See, for instance, Todd (2007). 5

7 follows. Let Y denote the outcome (wealth), T denote treatment status with T = 1 if the household has a loan and T = 0 otherwise, and Y 1 denote wealth if the household had a loan and Y 0 if it did not. Then for a given household the impact of having a loan would be Y 1 Y 0. Clearly, only observe one of either Y 1 or Y 0 is observed, since a household either has a loan or not, so that Y = (1 T )Y 0 + T Y 1 If loans were allocated randomly to households then the average difference in outcome between households with loans and those without would be an unbiased estimate of the average treatment effect, E(Y 1 Y 0 ). In the absence of random treatment allocation, this naïve estimator will yield biased results if treatment and control populations have systematic differences, as we would expect them to have in this case, especially since self-selection and other factors such as land ownership play a large role in obtaining a loan. This requires us to adjust the sample of households with and without loans suitably so as to make them comparable. Our interest is in the average treatment effect for those households which did obtain a loan, i.e. the ATT or average treatment effect for the treated, E(Y 1 Y 0 T = 1). Given the vector X of observed covariates, we assume that assignment to treatment is strongly ignorable (Rosenbaum and Rubin, 1983) if we surmise that the distribution of the counterfactual outcome Y 0 is independent of treatment status once X has been taken into account 5 : Y 0 T X From this it follows that E(Y 0 T = 1, X) = E(Y 0 T = 0, X) = E(Y 0 X). Provided that the probability that a household receives a loan is always less than one 6, P (T = 1 X) < 1), we can impute the missing Y 0 value for each 5 The more general assumption Y 1, Y 0 T X is required if we want to estimate the ATE or average treatment effect, since then we must impute both counterfactuals. 6 Again, the more general assumption required for estimating the ATE is 0 < P (T = 1 X) < 1. This, together with Y 1, Y 0 T X is the assumption of unconfoundedness (Rosenbaum and Rubin, 1983). 6

8 observed Y 1 and use this to estimate the ATT: E(Y 1 Y 0 T = 1, X) = E(Y 1 T = 1, X) E(Y 0 T = 1, X) = E(Y 1 T = 1, X) E(Y 0 T = 0, X) The idea behind matching approaches is to find, for each household with a loan, a control household without a loan that is the very similar in terms of X, and use the outcome of this control household as the imputed outcome for when the treated household had it not received the treatment. The differences in these pairs of treatment-control outcomes are then averaged across the sample to arrive at an estimate of the ATT. Ideally, we would compare households with and without loans who had identical values of X, but since this is seldom possible given that X is a multidimensional vector of covariates, instead, a metric is defined to measure the closeness of two different X vectors, and choose comparable or matched households according to this metric. Rosenbaum and Rubin (1983) showed that matching on X was equivalent to matching on the propensity score, the probability of a household receiving the treatment given its vector of covariates X, thus reducing a multidimensional matching problem to a single dimension, the propensity score. Since the true propensity score is not known, this is usually estimated by a logit or probit model, leading thus to a semiparametric matching process. More recently, Abadie and Imbens (2006) show that using the Mahalanobis distance as the metric gives rise to a non-parametric matching estimator that has desirable large-sample properties. Given two households with vectors X and X, the (squared) Mahalanobis distance is (X X ) T V 1 (X X ) where V 1 is the sample variance-covariance matrix. The Mahalanobis distance is thus similar to the Euclidean distance in R n but normalizes the contribution of each component of X by the inverse of the sample variance, and adds in cross-products normalized by the respective inverse covariances. This has the intuitively appealing property that poor matching on a covariate with large sample variance will be penalized less than poor match on covariate with 7

9 small variance. However, a potential drawback of the Mahalanobis distance measure is that binary variables that rarely take on the value 1 in the sample (for instance, if only a small part of the sample comes from a certain region) will get a large weight in the matching process because their variance is small. This might be desirable, but we might equally well suggest that matching on region is not of critical importance, in comparison to, say, the amount of land owned by a household. I employ matching estimators based on the propensity score (1-n, kernel and radius matching) as well as Mahalanobis distance 7. The criterion for choosing between various available matching methods is usually based on a comparison of the degree of balance in the resulting matched sets of covariates. In turn, there are different ways of evaluating balance. Ideally, the aim of checking balance is to compare the (multidimensional) distribution of covariates across treated and control groups in the matched data. This is unfeasible in general, and it is unclear even how to compare a two-dimensional set of covariates because, for instance, comparing two bivariate histograms is quite complex given there there will likely be several areas in the two-dimensional domain which will have zero density (Stuart, 2010). Comparing higher-dimensional sets of covariates is even more difficult, and balance checking is usually done for the marginal densities of this multi-dimensional distribution. As a result, most evaluations of balance are restricted to comparing treatment and control groups covariate by covariate. In the economics literature (Dehejia and Wahba (2002); Smith and Todd (2005); Todd (2007); Lechner et al. (2011) amongst others) balance is usually checked using t-tests for the equality of means of each covariate between treatment and control groups 8. In a recent paper, Imai et al. (2008) demonstrate that the use of t-tests and hypothesis tests in general for balance 7 Lechner et al. (2011) use the Mahalanobis distance based on a vector that includes the propensity score as well as covariates already used to calculate the propensity score to estimate the treatment effects of training programs 8 This is also used as a basis for determining the specification of the propensity score equation (Caliendo and Kopeinig, 2008) by iterating specifications systematically until a specification satisfying the balancing property, as defined used t-tests, is obtained. 8

10 checking is flawed, and by arbitrarily dropping part of the data they show that a t-test can be made both significant or insignificant 9. This is because the t statistic depends on the sample size since the theory for a t-test assumes that we are comparing samples drawn from an underlying hypothetical population, and thus significance varies with the size of the matched sample. Indeed, an insignificant t-test that could be interpreted as satisfying the balance criteria could simply be a result of a smaller sample size, since several observations are discarded as a result of matching. Instead, the criterion used to judge balance should consider the matched sample as the population of interest, and be independent of sample size. In any case, a t-test only serves to compare means and is uninformative about potential differences in higher moments of the covariate distributions. Imai et al. (2008); Ho et al. (2007); Stuart (2010); Austin (2008) therefore recommend measures that are independent of sample size to judge balance after matching. These include standardized bias plots for each covariate used to estimate the propensity score, as well as quantile-quantile plots for matched treatment and control samples and their summary measures such as the mean and maximum deviation from the 45 line of identical distribution. Box plots based on the standardized bias are also used as summary measures of the extent of balance in matched samples. The standardized bias before and after matching is defined by Rosenbaum and Rubin (1985) as follows (see, also, the discussion in Lee (2011)): SB u x = 100( X T X C ) s 2 (x t)+s 2 (x c) 2 and SB m x = 100( X T,M X C,M ) s 2 (x t)+s 2 (x c) 2 9 They conclude that if a comparison of means for a given covariate passes a t-test, that is, the t-statistic is insignificant, then this meaningless since the underlying covariate distributions may indeed be balanced (as the t-test indicates) or they may not, and the t-test is actually uninformative about the true balance, rather than being consistently overly-conservative or overly-liberal. 9

11 where SB u x denotes the standardized bias for covariate x in the unmatched sample, and SB m x denotes the corresponding quantity in the matched sample. s 2 (x t ) denotes the squared sample standard deviation, or sample variance, for the treatment sample of covariate x, and likewise s 2 (x c ) denotes the same for the control sample. The standardized bias is thus a dimension-free quantity, and can be used to judge the amount of imbalance that remains following any matching exercise. However the standardized bias is a measure that only summarizes the means for each covariate and ignores higher moments, whereas, quantile-quantile plots allow for comparing the overall distribution of a pair of random variables (Imai et al., 2008; Stuart, 2010). Stuart (2010) suggests computing two quantities based on a quantile-quantile plot the average and maximum deviation of the plot from the 45 line of equality, and computing how much these have improved in the matched data compared to the unmatched data. 3 Data and definition of estimation sample We use data from the latest Debt and Investment Survey carried out by the National Sample Survey Organisation that were collected in 2003 and are nationally representative 10, and restrict the sample to farmers living in rural areas in the ten largest states 11. The survey took place across two visits a few months apart, and collected data on a large set of household demographics, the land and other assets owned by the household, as well as its credit transactions in the recent past. The former includes caste, household size and composition in terms of age and gender, primary occupation, education and marital status, while assets include consumer durables and farming-related assets. The first visit was carried out between January and August 2003, gathered 10 Round 59, Schedule Largest in terms of population. These states are Andhra Pradesh, Bihar, Gujarat, Karnataka, Madhya Pradesh, Maharasthra, Rajasthan, Tamil Nadu and West Bengal. 10

12 Figure 1: Timeline of survey data on all variables, and in particular, asked households to list what consumer durable assets and loans they had as of June 2002, and what transactions in these they had undertaken since then. The second follow-up visit with the same household was undertaken between August and December The survey questionnaire for the second visit is a small subset of the first visit s questionnaire, but importantly, asked households to detail what consumer durable assets and loans they had as of June 2003, and again, what transactions they undertook between this date and the date of the survey. The data for consumer durable assets and loans thus exist for two dates exactly a year apart: June 2002 and June The data on loans include their source, amount and purpose. Consumer durables include furniture, utensils, electronic gadgets and clothing. The timeline of the survey questionnaire is summarized in figure 1 we use June 2002 and June 2003 as the reference dates since these a) provide a gap of one year, which is the longest beforeafter period there are data for, and b) these provide a uniform reference period for each household, unlike the actual survey dates which vary over January to August (first visit) and September to December (second visit). In order to credibly measure the treatment effect in an observational study it 11

13 is essential for the covariates used to balance the data to be independent of treatment status, or for the data on them to have been collected before the treatment was allocated (Cochran (1965); see also the discussion in Rosenbaum (2002)). The treatment here is the act of obtaining an agricultural loan from a formal source primarily banks, while the outcome variable is an index of consumer durable assets constructed using principal components analysis, which, as I discuss, is a useful proxy for households wealth status. To obtain a coherent estimate the treatment effect, I therefore restrict the estimation to those households that did not have a loan in June 2002, and define treatment status according to whether a household subsequently obtained a loan between June 2002 and June 2003, and correspondingly, control households to be those that did not obtain a loan during this period. And, in order to avoid the potentially confounding influence of loans from sources other than banks primarily moneylenders as well as bank loans meant for consumption rather than agricultural production, I drop households who have such loans at either date. The final sample thus consists of 9,778 households. Of these, 1,697 or roughly 17% obtained a loan between June 2002 and June 2003, yielding 1,697 treatment and 8,081 control observations. The outcome variable used to estimate treatment effects is an index of consumer durable assets owned by the household. While cash income might be preferable since it would be more sensitive to changes in agricultural earnings, the survey does not attempt to measure household income, owing perhaps to the twin challenges of assigning monetary values to non-cash earnings in kind or otherwise, and also, eliciting an honest response to questions about cash earnings. An alternative measure of income or wealth is consumption expenditure, but this is notorious for being vulnerable to recall errors which can induce biases of indeterminate sign and vary with the recall period (Sundaram and Tendulkar, 2003; Deaton and Kozel, 2005). It is also insensitive to changes in wealth for rich households, and tends to fluctuate over the year, making it tenuous to use as an indicator of long or medium-term income and welfare. Finally, like cash income, the value of consumption expenditure is sensitive to price levels that potentially diverge across regions. 12

14 Therefore, I construct an index of households wealth status or long-run income using data on their ownership of consumer durables, using principal components analysis which calculates weights for each category of asset that are proportional to the variance in ownership for that asset (Sahn and Stifel, 2003; Filmer and Pritchett, 2001) 12. Using consumer durable assets to create an index results in a metric that is relatively robust to many of the problems outlined above, including recall error, inaccurate measurement, and sensitivity to prices. I construct two values of this index corresponding to consumer durables owned by the household in June 2002 and then a year later, in June The June 2002 value of the index is used as one of several covariates to match treatment and control households, while the June 2003 value of the index is used as the outcome variable. From here on I refer to this as the assets index as of June 2002 or June Since the assets index is in effect a ranking of households in terms of their wealth status, it has no direct economic interpretation. Consequently, treatment effects are interpreted in terms of the percentile shift in wealth status that a household would experience as a result of having or not having an agricultural loan. The other covariates used to match households are the land area owned by the household, an index of farm assets such as machinery and implements (again using principal components analysis) and demographic variables that include caste 13, household size, the proportion of children, adult males and adult females, the household head s age and education level and a dummy for 12 Hence, if households differ a lot in the number of televisions owned but not so much in footwear, then this technique will assign a higher weight to television ownership and less to footwear. 13 These are Scheduled Caste (SC), the lowest caste-group, Other Backward Castes (OBC), a caste-group of middling disadvantage, and Others, the high castes. I drop households whose caste is Scheduled Tribes (ST) from the sample since ST is not actually a caste, but the Government of India uses it as as a part of the caste-classification system since STs are disadvantaged and likely to be poorer than other peoples. Likewise, the areas where STs are in majority are likely to be systematically different and poorer than others. Part of these differences might be picked up by dummies for states since certain states have higher concentrations of tribal population, but one could argue that any estimation of the benefits (or otherwise) from loans for tribal households should take place separately from non-tribal households. 13

15 whether a married adult son lives in the household, together with two regionspecific variables; the dominant caste in the district in terms of aggregate land ownership, and finally, the state where the household resides. The data on these household attributes were collected during the first survey visit, and it is unlikely that their values would have changed between June 2002 and the actual date of survey since these variables are either independent of treatment and invariant to time (region-specific variables, household head s education and caste) or, would be independent of treatment but with small changes over time, in the case of age-related variables such as age of household head which would most likely have been one year less in June 2002 than they were on the date of the survey, which can be reasonably expected to wield negligible influence on the results. In the case of land area owned by the household, I verify that no land purchases have been made by the household between June 2002 and the (first) date of the survey, ensuring that the value of this covariate on the first survey visit is the same as it was in June 2002, and in particular, is unrelated to treatment status. Table 1 provides summary statistics for all the variables used, for the sample as well as by treatment status. 4 Results I begin by estimating the propensity score through a logit model to model allocation to treatment whether the household obtained a loan between June 2002 and June 2003 or not. I then use different matching algorithms to construct matched sets of treatment and control observations, and compare the remaining imbalance in each matched set in order to choose the best matching algorithm(s) as that which results in the least amount of imbalance. The variables used in the propensity score equation include dummies for the state in which the household resides and the dominant caste in that district together with the following household-level variables: area owned by 14

16 Table 1: Summary statistics of variables by treatment status Sample Treatment Control Variable Mean SD Mean SD Mean SD Continuous variables Categorical variables Assets index June Assets index June Land area HH head education (yrs) HH head age (yrs) HH size Propn children Propn adult males Propn adult females Farm assets index Caste SC Caste OBC Caste Others Dominant SC Dominant OBC Dominant Others Live with married son Andhra Pradesh Bihar Gujarat Karnataka Madhya Pradesh Maharashtra Rajasthan Tamil Nadu Uttar Pradesh West Bengal

17 Figure 2: Kernel density plot of the estimated propensity score Density Estimated propensity score Treatment observations Control observations the household, assets index as of June 2002, age and years of education of the household head, proportions of children, adult males and adult females in the household, farm assets (index) owned by the household as of June 2002, the household s caste, and a dummy for whether a married son lives in the household. This specification for the propensity score equation was arrived at by iteratively adding variables, and keeping those which were significant. Higher order terms for land area owned by the household and the age of the household head were also included since these were also statistically significant. There is good reason to expect land area owned to be the most important determinant of treatment status, since land is used as a collateral to obtain credit, and, given that land is the prerequisite for agricultural activity, it is also likely to influence demand for credit. Table 2 contains the results of the logit regression used to estimate the propensity score, and figure 2 shows a kernel density plot of the estimated propensity score by treatment status. There is substantial overlap between the propensity scores of treatment and control households, indicating that 16

18 Table 2: Estimation of the propensity score using a logit model Regressor Coefficient Standard error Assets June (0.0203) Land area (0.0341) Land area sq ( ) Land area cube ( ) HH head edu (yrs) (0.0195) HH head edu sq ( ) HH head age (yrs) (0.0133) HH head age sq ( ) Live with married son (0.0862) Farm assets (0.0203) Propn of children (0.194) Propn of adult males (0.232) Propn of adult females (0.301) HH size (0.0126) Caste OBC (0.0873) Caste Others (0.0935) OBC dominated (0.314) Others dominated (0.311) AP (0.145) Bihar (0.154) Gujarat (0.183) Karnataka (0.168) Madhya Pradesh (0.147) Maharashtra (0.140) Rajasthan (0.169) Tamil Nadu (0.179) Uttar Pradesh (0.120) Constant (0.481) N 9777 ll Standard errors in parentheses p < 0.05, p < 0.01, p <

19 matching is likely to be feasible. We use nearest-neighbour, radius, kernel and Mahalanobis matching. nearest-neighbour matching, each treatment observation is matched with the nearest n controls, where nearness is defined as the absolute difference in propensity scores between the respective observations. I use the nearest 1, 3 and 5 neighbours. In general, the fewer the number of neighbours used in matching the lower we can expect the bias to be since the matches will tend to be of higher quality. However, as more neighbours are used for matching the estimates of treatment effect tend to be more efficient and also make better use of the large set of available controls. Throughout, matching is done with replacement since this allows a better matches to be used more than once, thereby reducing bias (Dehejia and Wahba, 2002), and matching is restricted to the region of common support on the estimated propensity score. And, the 5% of treatment cases for whom the corresponding density of the control cases is thinnest are dropped, to restrict matches to the thicker regions of the overlapping propensity scores. Further, matching is performed using calipers, or a maximum permissible difference in the propensity score of a given treatment-control pair in order to avoid poor matches. I follow standard practice and put calipers equal to one fifth of a standard deviation of the estimated propensity score. In radius matching (Dehejia and Wahba, 2002), each treatment case is matched with all available controls that lie within a specified radius in terms of the propensity score, potentially making better use of available controls compared to nearest-neighbour methods. In kernel matching, a kernel function is centered on each treatment case and used to weight available controls. Radius matching is thus kernel matching with a uniform kernel, but kernels such as the Epanechnikov and Gaussian give higher weight to controls that are closer to the given treatment case, and vice-versa An additional method of matching is that of stratification on the propensity score. This has the advantage that all available controls are used provided they are on the common support of the propensity score, but we can expect the quality of matches to be poorer in general than matching within calipers whenever the calipers used are significantly In 18

20 Finally, Mahalanobis matching matches observations based on the Mahalanobis distance between each pair of treatment and control observations. Similar to nearest-neighbour matching, I again use 1, 3 and 5 nearest Mahalanobis neighbours to match on. I use the same set of variables that were used to estimate the propensity score to calculate the Mahalanobis distance, and also include the estimated propensity score as one of the variables Evaluating balance Figure 3 plots the standardized bias for each variable used in defining the propensity score before and after matching while 4 summarizes these in box plots. For clarity 1-3 and 1-5 nearest neighbour matching, and 1-3 and 1-5 Mahalanobis matching are excluded since their results are quite similar. There is substantial imbalance in the original sample, and all the matching methods succeed in reducing this imbalance substantially, even though Mahalanobis matching reduces imbalance to a lesser degree compared to the other methods for certain variables: assets, land area owned, education of the household head, and farm assets. As discussed, land area owned is an important variable to balance on, and the substantial imbalance that results for this variable using Mahalanobis matching signals the inappropriateness of this matching method. Summary plots of the standardized bias are shown in figure 4, but these are of limited use in comparing matching methods. Nearest neighbour, radius and kernel matching all appear to perform very similarly, even though Mahalanobis performs worse. Standardized bias summarizes balance only in terms of the mean for each covariate and ignores higher moments or indeed the distribution in general. In order to judge the entire distribution for each covariate, quantile-quantile plots are used to compare the distribution for a given covariate between treatsmaller than the propensity score strata. 15 The extent of covariate balance remains all but unchanged if the propensity score is excluded 19

21 Standardized Figure 3: Standardized bias of bias variables plots before before and and after after matching matching Region Household characteristics Propensity score Assets June 2002 Land area Land area sq Land area cube Edu HH head Edu HH head sq Age HH head Age HH head sq HH size Propn children Propn adult males Propn adult females Farm assets Caste SC Caste OBC Caste Others Dominant SC Dominant OBC Dominant Others live with married son Andhra Pradesh Bihar Gujarat Karnataka Madhya Pradesh Maharashtra Rajasthan Tamil Nadu Uttar Pradesh West Bengal Unmatched sample 1-1 matching Radius matching Kernel matching Mahalanobis matching Standardized bias 20

22 Figure 4: Summary box plots of standardized bias Standardized bias across all variables Matching method standardized bias Unmatched 1-1 Nearest neighbour 1-3 Nearest neighbours 1-5 Nearest neighbours Radius Kernel 1-1 Mahalanobis 1-3 Mahalanobis 1-5 Mahalanobis 21

23 ment and control samples before and after matching. The quantile-quantile plot will be a straight line inclined at 45 if and only if the two variables used on the respective axes are identically distributed. In the current exercise, we are interested in comparing the distribution of each covariate in the treatment and matched control groups. Thus, if the matched sets are perfectly balanced, the quantile-quantile plots of treatment against control samples will be a straight line. In practice, balancing never succeeds entirely and so the extent of deviation from this straight line indicates the remaining extent of imbalance. This can be judged visually, and also through the summary measures of average and maximum deviation from this line. Table 3 summarizes the mean deviation of the quantile-quantile plots from the 45 line to compare, for each covariate, the distributions of the treatment and control samples before and after matching. In order to construct the quantile-quantile plots, matched sets of treatment and control observations must be constructed after taking into account the weights obtained from the matching process (Joffe et al., 2004). I do this by expanding the matched dataset such that the number of times a given observation is replicated is proportional to the weights placed on it by the matching process. Since weights can be both fractions as well as integers, all the weights are first multiplied by a suitably large integer, the result is rounded off to the nearest integer, and these rounded (integer) weights are then used as frequency weights to expand the data 16. Unmatched observations receive a weight of zero, and are therefore dropped. Figure 5 shows quantile-quantile plots for 1-1 nearest neighbour matching and table 3 provides summary measures of balance for this and other matching methods, for each continuous covariate. Nearest neighbour matching appears to be the most successful in removing imbalance, though the reductions in imbalance are quite similar for one, three and five nearest-neighbour matching. 1-1 matching shows similar improvement in land area owned com- 16 For example, if the weights present in the data are 1/5, 1/3 and 1 and we multiply each weight by 10, then this procedure will yield a new dataset with the corresponding observations duplicated 2, 3 and 10 times respectively 22

24 pared to 1-3 and 1-5 matching, but more improvement for the corresponding higher order terms square and cube though it is less successful in balancing the assets index, household size, and education level of the household head compared to 1-3 and 1-5 matching. Column 4 of of table 3 shows that radius matching does lead to substantial gains in covariate balance for the continuous covariates, but though similar, these gains are smaller than those obtained using nearest neighbour matching. Likewise, column 5 shows the improvement in covariate balance that obtains from kernel matching using an Epanechnikov kernel and a bandwidth equal to 10% of a standard deviation of the propensity score 17,18, which again leads to higher imbalance for most covariates compared to nearest neighbour matching. Columns 6-8 in table 3 summarize the balance that obtains from Mahalanobis matching. This matching uses calipers corresponding to the 95th percentile of all nearest-neighbour Mahalanobis distances, thus discarding certain very large distances that signify poor matches. Even so, the decrease in mean deviation for the quantile-quantile plots is modest compared to the other techniques, and the extent of imbalance increases as we match with additional neighbours 19 Tables 4 provides quantile-quantile summary measures for continuous variables for 1-5 nearest neighbour matching. The results for 1-1 and 1-3 nearest neighbour matching are quite similar and are therefore not shown. While 17 We tried several specifications using Epanechnikov and Gaussian (normal) kernels, with kernel bandwidths of 0.01, 0.02 and 0.04 (corresponding to 10%, 20% and 40% of the standard deviation of the estimated propensity score). Mean deviations from the 45 line of equality for each covariate improve with smaller bandwidth as we might expect, since poorer matches are excluded (or given a lower weight with a Gaussian kernel) 18 Compared to a Gaussian kernel, the Epanechnikov kernel leads to better balance on land area owned, assets, education and age of the household head, while the Gaussian kernel leads to better balance on the proportion of children, adult males and adult females but these differences are very small 19 We performed Mahalanobis matching with and without using the propensity score as a covariate in calculating the Mahalanobis distance, and also by excluding the higher order terms used to estimate the propensity score, however all specifications perform similarly and the resulting imbalance is significantly higher than for the other matching methods. 23

25 Table 3: Relative reductions in imbalance for different matching methods Nearest neighbour Radius Kernel Mahalanobis Variable Assets index June Land area Land area sq Land area cube HH head education (yrs) HH head education sq HH head age (yrs) HH head age sq HH size Propn children Propn adult males Propn adult females Farm assets index controls used Notes Percentage reduction in mean deviation from the (45 ) line of equality in quantile-quantile plots by matching method. Only continuous variables are shown since quantile-quantile plots are relevant only for these. 24

26 maximum deviation does not always improve, and actually deteriorates for certain variables, mean deviations improve by a large extent for every variable, and the ratio of difference in means to the standard deviation in matched samples is quite small, indicating that matching succeeds in reducing imbalance by a large extent. Table 5 summarizes balance in binary variables, showing the respective proportions of each variable in treatment and control samples before and after matching. Again, the gains in balance are substantial and the absolute difference in proportions after matching (column 5) are very small. 4.2 Treatment effects Our focus is on measuring the average treatment effect for the treated (ATT), i.e., for those households who received a loan. Each matching method has enabled us to construct a counterfactual by choosing households without loans who are as similar as possible to a given household who was treated, i.e. had a loan, even as Mahalanobis matching results in rather more imbalance remaining in the matched data compared to the other methods. The outcome variable is an index of consumer durable assets owned by the household on June 2003 which allows us to rank households in terms of their wealth status. The ATT is therefore interpreted as a shift in percentiles of this asset index, or in other words, the jump up or down in the rank of a given household that took place on account of that household having a loan. As a benchmark for comparison, table 6 presents the simple difference and regression-adjusted estimates of the treatment effect, where the variables used in the regression adjustment are those used to estimate the propensity score. Table 7 shows the estimates of ATT for each matching method in terms of the assets index and also the percentile shift this ATT translates into for a household at the mean and median of the assets index. For nearest-neighbour, radius and kernel matching, the ATT is calculated as the average difference in outcome between treatment and control households, 25

27 Table 4: Summary measures of balance for continuous variables using quantile-quantile plots Variable Mean Improvement (%) Maximum Improvement (%) SD ratio a Assets index June Land area Land area sq Land area cube e HH head education (yrs) HH head education sq HH head age (yrs) HH head age sq HH size Propn children Propn adult males Propn adult females Farm assets index Notes Mean and maximum refer, respectively, to the mean and maximum deviation from the 45 line of equality in quantile-quantile plots of treatment versus control groups before and after 1-5 nearest neighbour matching. a Ratio of standard deviation to the difference in means of treatment and control groups in the matched sample. 26

28 Table 5: Summary measures of balance for binary variables Unmatched proportions Matched proportions Variable Control Treatment Control Treatment Difference a Caste SC Caste OBC Caste Others Dominant SC Dominant OBC Dominant Others live with married son Andhra Pradesh Bihar Gujarat Karnataka Madhya Pradesh Maharashtra Rajasthan Tamil Nadu Uttar Pradesh West Bengal Region Notes Proportions of categorical variables before and after 1-5 nearest neighbour matching. a Difference in proportions for treatment and control groups in the matched sample. 27