A Simple Poverty Scorecard for Bangladesh

Transcription

1 A Simple Poverty Scorecard for Bangladesh Shiyuan Chen and Mark Schreiner 24 April 2009 This document and related tools are at Abstract This study uses the 2005 Bangladesh Household Income and Expenditure Survey (HIES) to construct an easy-to-use scorecard that estimates the likelihood that a household has expenditure below a given poverty line. The scorecard uses ten simple indicators that field workers can quickly collect and verify. Poverty scores can be computed on paper in the field in about five to ten minutes. The scorecard s accuracy and precision are reported for a range of poverty lines. The poverty scorecard is a practical way for pro-poor programs in Bangladesh to monitor poverty rates, track changes in poverty rates over time, and target services. Acknowledgements This paper was funded by the Microcredit Summit Campaign. It revises and expands a 2006 paper that used the 2000 HIES and that was funded by the Grameen Foundation. Special thanks go to Sajjad Zohir for acquiring the survey data from the Bangladesh Bureau of Statistics. Nobuo Yoshida of the World Bank generously provided household expenditure, national poverty lines by stratum for 2005, survey sampling weights, and price deflators. Thanks also go to Faizuddin Ahmed, Nigel Biggar, Sam Daley-Harris, Robert Driscoll, Brian McConnell, DSK Rao, Sangita Sigdyal, Salman Zaidi, and Hassan Zaman. Authors Shiyuan Chen is Senior Analyst with Microfinance Risk Management, L.L.C., 2441 Tracy Avenue, Kansas City, MO , U.S.A., +1 (816) , shiyuan@microfinance.com. Mark Schreiner is the Director of Microfinance Risk Management, L.L.C., mark@microfinance.com. He is also Senior Scholar, Center for Social Development, Washington University in Saint Louis, Campus Box 1196, One Brookings Drive, Saint Louis, MO , U.S.A.

2 Figure 1: A simple poverty scorecard for Bangladesh Entity Name ID Date (DD/MM/YY) Member: Joined: Loan officer: Today: Branch: Household size: Indicator Value Points Score 1. How many household members A. Four or more 0 are 11-years-old or B. Three 9 younger? C. Two 12 D. One 19 E. None Does any household member A. Yes 0 work for a daily wage? B. No What type of latrine does the household use? A. Open field 0 B. Kacha latrine (temporary or permanent), pacca (pit or 5 water seal), or sanitary 4. How many rooms does the A. One, two, or three 0 household occupy B. Four 7 (excluding rooms used for business)? C. Five or more What is the main construction A. Mud brick, hemp/hay/ material of the walls? bamboo, or other 0 B. C.I. sheet/wood 2 C. Brick/cement 8 6. What is the main construction A. Tile/wood, hemp/hay/ material of the roof? bamboo, or other 0 B. C.I. sheet/wood 2 C. Cement What is the total cultivable A. None, or 0.5 acres or less 0 agricultural land owned B. More than 0.5 acres, and by the household? 1.0 acres or less 4 C. More than 1.0 acres 6 8. Does the household own a A. No 0 television? B. Yes 7 9. Does the household own a A. No 0 two-in-one cassette B. Yes Does the household own a A. No 0 wristwatch? B. Yes 4 Microfinance Risk Management, L.L.C., Score:

3 Figure 1: A simple poverty scorecard for Bangladesh (no points) Entity Name ID Date (DD/MM/YY) Member: Joined: Loan officer: Today: Branch: Household size: Indicator 1. How many household members are 11-yearsold or younger? 2. Does any household member work for a daily wage? 3. What type of latrine does the household use? 4. How many rooms does the household occupy (excluding rooms used for business)? 5. What is the main construction material of the walls? 6. What is the main construction material of the roof? 7. What is the total cultivable agricultural land owned by the household? 8. Does the household own a television? 9. Does the household own a two-in-one cassette 10. Does the household own a wristwatch? A. Four or more B. Three C. Two D. One E. None A. Yes B. No Value A. Open field B. Kacha latrine (temporary or permanent), pacca latrine (pit or water seal), or sanitary A. One, two, or three B. Four C. Five or more A. Mud brick, hemp/hay/bamboo, or other B. C.I. sheet/wood C. Brick/cement A. Tile/wood, hemp/hay/bamboo, or other B. C.I. sheet/wood C. Cement A. None, or 0.5 acres or less B. More than 0.5 acres, and 1.0 acres or less C. More than 1.0 acres A. No B. Yes A. No B. Yes A. No B. Yes Microfinance Risk Management, L.L.C., 1

4 A Simple Poverty Scorecard for Bangladesh 1. Introduction This paper presents an easy-to-use poverty scorecard that pro-poor programs in Bangladesh can use to estimate the likelihood that a household has expenditure below a given poverty line, to monitor groups poverty rates at a point in time, to track changes in groups poverty rates between two points in time, and to target services to households. The direct approach to poverty measurement via surveys is difficult and costly, asking households about a lengthy list of expenditure categories such as What was the value of firewood consumed that was bought in cash/credit or wages in-kind? What was the value of firewood consumed that was produced by the household or received? What was the sum of them?... ). In contrast, the indirect approach via poverty scoring is simple, quick, and inexpensive. It uses ten verifiable indicators (such as What is the main construction material of the walls? or Does the household own a television? ) to get a score that is highly correlated with poverty status as measured by the exhaustive survey. The poverty scorecard here differs from proxy means tests (Coady, Grosh, and Hoddinott, 2002) in that it is tailored to the capabilities and purposes not of national governments but rather of local, pro-poor organizations. The feasible povertymeasurement options for these organizations are typically subjective and relative (such 1

5 as participatory wealth ranking by skilled field workers) or blunt (such as rules based on land-ownership or housing quality). These approaches may be costly, their results are not comparable across organizations nor across countries, and their accuracy and precision are unknown. Suppose an organization wants to know what share of its participants are below a poverty line; for example, it might want to report using the USD1.25/day poverty line at 2005 purchase-power parity for the Millennium Development Goals. Or it might want to report how many participants are among the poorest half of people below the national poverty line (as required of USAID microenterprise partners). Or suppose an organization wants to measure movement across a poverty line (for example, to report to the Microcredit Summit Campaign). In all these cases, the organization needs an expenditure-based, objective tool with known accuracy. While expenditure surveys are costly even for governments, many small, local organizations can implement an inexpensive scorecard that can serve for monitoring, management, and targeting. The statistical approach here aims to be understood by non-specialists. After all, if managers are to adopt poverty scoring on their own and apply it to inform their decisions, they must first trust that it works. Transparency and simplicity build trust. Getting buy-in matters; proxy means tests and regressions on the determinants of poverty have been around for three decades, but they are rarely used to inform decisions. This is not because they do not work, but because they are presented (when they are presented at all) as tables of regression coefficients incomprehensible to non- 2

6 specialists (with cryptic indicator names such as LGHHSZ_2, negative values, and many decimal places). Thanks to the predictive-modeling phenomenon known as the flat max, simple scorecards are about accurate as complex ones. The technical approach here is also innovative in how it associates scores with poverty likelihoods, in the extent of its accuracy tests, and in how it derives formulas for standard errors. Although these techniques are simple and standard in the for-profit field of credit-risk scoring, they have rarely or never been applied to poverty scorecards. The scorecard (Figure 1) is based on the 2005 Household Income and Expenditure Survey (HIES) conducted by the Bangladesh Bureau of Statistics (BBS). Indicators are selected to be: Inexpensive to collect, easy to answer quickly, and simple to verify Strongly correlated with poverty Liable to change over time as poverty status changes All points in the scorecard are non-negative integers, and total scores range from 0 (most likely below a poverty line) to 100 (least likely below a poverty line). Nonspecialists can collect data and tally scores on paper in the field in five to ten minutes. Poverty scoring can be used to estimate three basic quantities. First, it can estimate a particular household s poverty likelihood, that is, the probability that the household has per-capita expenditure below a given poverty line. Second, poverty scoring can estimate the poverty rate of a group of households at a point in time. This is simply the average poverty likelihood among the households in the group. 3

7 Third, poverty scoring can estimate changes in the poverty rate for a given group of households (or for two independent representative samples of households from the same population) between two points in time. This estimate is the change in the average poverty likelihood of the group(s) of households over time. Poverty scoring can also be used for targeting services. To help managers choose a targeting cut-off, this paper reports several measures of targeting accuracy for a range of possible cut-offs. This paper presents a single scorecard (Figure 1) whose indicators and points are derived from household expenditure data and the USD1.25/day/person 2005 PPP poverty line. Scores from this scorecard are calibrated to poverty likelihoods for six poverty lines. The scorecard is constructed and calibrated using a sub-sample of the data from the 2005 HIES. Its accuracy is validated on a different sub-sample from the 2005 HIES as well as on the entire 2000 HIES. 1 While all three scoring estimators are unbiased when applied to the population from which they were derived (that is, they match the true value on average in repeated samples from the same population from which the scorecard was built), they are like all predictive models biased to some extent when applied to a different population. 2 1 Accuracy is not tested with the 1991/2 and 1995/6 Household Expenditure Surveys because they lack many indicators in the scorecard constructed from the 2005 HIES. 2 Examples of different populations include a nationally representative sample at another point in time or a non-representative sub-group (Tarozzi and Deaton, 2007). 4

8 Thus, while the indirect scoring approach is less costly than the direct survey approach, it is also always biased in practice. (The direct survey approach is unbiased by definition.) There is bias because scoring must assume that the future relationship between indicators and poverty will be the same as in the data used to build the scorecard as well as the same in all sub-groups as it is in the population. 3 Of course, this assumption ubiquitous and inevitable in predictive modeling holds only partly. When applied to the 2005 validation sample for Bangladesh with n = 16,384, the difference between scorecard estimates of groups poverty rates and the true rates at a point in time is percentage points for the upper national poverty line, and the average absolute difference is 0.7 percentage points across all six lines. Because the 2005 validation sample is representative of the same population as the data that was used to construct the scorecard and because all the data comes from the same time frame, the scorecard estimators are unbiased and any differences are due to sampling variation; the average difference would be zero if the whole 2005 HIES were to be repeatedly redrawn and divided into sub-samples before repeating the entire scorecard-building and accuracy-testing process. For n = 16,384, the 90-percent confidence intervals for these estimates are +/ 0.5 percentage points or less for estimates of a poverty rate at a point in time for the 3 Bias may also result from changes in the quality of data collection, from changes over time in the real value of poverty lines, from imperfect adjustment of poverty lines to account for differences in cost-of-living across time or geographic regions, or from sampling variation across expenditure surveys. 5

9 2005 validation sample. For n = 1,024, these intervals are +/ 2.1 percentage points or less. When the scorecard built from the 2005 construction and calibration samples is applied both to the 2005 validation sample and to the entire 2000 HIES with n = 16,384, the difference between scorecard estimates and true values for changes in groups poverty rates is 1.3 percentage points for the upper national line. While the true change was 9.2 percentage points, the scorecard estimates a change of 10.5 percentage points. Across all six lines, the average estimated change is about 10 percent too big. For n = 16,384, the 90-percent confidence intervals for these estimates of change are +/ 0.8 percentage points or less Section 2 below describes data and poverty lines. Section 3 places the new scorecard here in the context of existing exercises for Bangladesh. Sections 4 and 5 describe scorecard construction and offer practical guidelines for use. Sections 6 and 7 detail the estimation of households poverty likelihoods and of groups poverty rates at a point in time. Section 8 discusses estimating changes in poverty rates, and Section 9 covers targeting. The final section is a summary. 6

10 2. Data and poverty lines This section discusses the data used to construct and test the poverty scorecard. It also presents the poverty lines to which scores are calibrated. 2.1 Data The scorecard is based on data from the 10,080 households in the 2005 HIES. This is the best, most recent national expenditure survey available for Bangladesh. Households are randomly divided into three sub-samples (Figure 2): Construction for selecting indicators and points Calibration for associating scores with poverty likelihoods Validation for testing accuracy on data not used in construction or calibration In addition, the 2000 HIES is used in the validation of estimates of changes in poverty rates for two independent samples between two points in time. 2.2 Poverty rates and poverty lines Rates As a general definition, the poverty rate is the share of people in a given group who live in households whose total household expenditure (divided by the number of members) is below a given poverty line. Beyond this general definition, there two special cases, household-level poverty rates and person-level poverty rates. With household-level rates, each household is 7

11 counted as if it had only one person, regardless of true household size, so all households are counted equally. With person-level rates (the head-count index ), each household is weighted by the number of people in it, so larger households have greater weight. For example, consider a group of two households, the first with one member and the second with two members. Suppose further that the first household has per-capita expenditure above a poverty line (it is non-poor ) and that the second household has per-capita expenditure below a poverty line (it is poor ). The household-level rate counts both households as if they had only one person and so gives a poverty rate for the group of 1 (1 + 1) = 50 percent. In contrast, the person-level rate weighs each household by the number of people in it and so gives a poverty rate for the group of 2 (1 + 2) = 67 percent. Whether the household-level rate or the person-level rate is relevant depends on the situation. If an organization s participants include all the people in a household, then the person-level rate is relevant. Governments, for example, are concerned with the well-being of people, regardless of how those people are arranged in households, so governments typically report person-level poverty rates. If an organization has only one participant per household, however, then the household-level rate is relevant. For example, if a microlender has only one borrower in a household, then it might want to report household-level poverty rates. Based on Bangladesh s 2005 and 2000 HIES, this paper reports poverty rates and poverty lines by 2005 stratum at both the household level and the person level (Figure 8

12 3). The poverty scorecard is constructed using the 2005 HIES and household-level lines, scores are calibrated to household-level poverty likelihoods, and accuracy is measured for household-level rates. This use of household-level rates reflects the belief that they are the most relevant for most pro-poor organizations. Organizations can estimate person-level poverty rates by taking a household-sizeweighted average of the household-level poverty likelihoods. It is also possible to construct a scorecard based on person-level lines, to calibrate scores to person-level likelihoods, and to measure accuracy for person-level rates, but it is not done here Poverty lines Bangladesh has two national poverty lines. For the country as a whole in 2005, the national upper (lower) line corresponds with a household-level poverty rate of 37.2 (23.1) percent and a person-level poverty rate of 40.0 (25.1) percent (Figure 3). At the household level from 2000 to 2005, poverty rates fell by 9.2 percentage points (upper line, Figure 2) and 8.9 percentage points (lower line). Because local pro-poor organizations may want to use different or various poverty lines, this paper calibrates scores from its single scorecard to poverty likelihoods for six lines: Upper national Lower national USAID extreme USD1.25/day 2005 PPP USD1.75/day 2005 PPP USD2.50/day 2005 PPP 9

13 The upper and lower national lines by 2005 stratum come from Nobuo Yoshida of the World Bank (Figure 16). The USAID extreme line is defined as the median expenditure of people (not households) below the national line (U.S. Congress, 2002). The scorecard here is constructed using the USD1.25/day line (2005 PPP). This is derived from: 2005 PPP exchange rate for individual consumption expenditure by households (International Comparison Project, 2008): Taka per $1.00 Price deflators from Nobuo Yoshida of the World Bank: 1.00 for 2005 and 0.77 for 2000 Using the formula in Sillers (2006), the USD1.25/day 2005 PPP lines for Bangladesh in 2005 and 2000 are: CPI ( 2005 PPP exchange rate) USD1.25 CPI Taka USD1.25 = Taka USD CPI ( 2005 PPP exchange rate) USD1.25 CPI Taka25.49 USD = Taka USD = = The USD1.75/day and USD2.50/day 2005 PPP lines are multiples of the USD1.25/day 2005 PPP lines. 10

14 The 2005 PPP lines just presented apply to Bangladesh as a whole. These are adjusted here for regional differences in cost-of-living as implicitly reflected in the upper national poverty lines (Figure 16). This is done using: L, a given national-level poverty line p i, population proportions by stratum (i = 1 to 16) π i, upper national poverty lines by stratum The stratum cost-of-living-adjusted poverty line L i for region i is then: L i = 16 L π i. p π j = 1 j j The given all-bangladesh poverty line L is the person-weighted average of the 16 stratum lines L i, with the differences in the stratum lines reflecting regional differences in the cost of living. 11

15 3. The context of poverty scorecards for Bangladesh This section discusses existing scorecards in terms of their goals, methods, poverty lines/benchmarks, indicators, accuracy, precision, and cost. There are at least eight existing poverty scorecards for Bangladesh; why one more? First, estimates from the scorecard here are tested out-of-sample, and accuracy, precision, and formulas for sample size and standard errors are reported. Second, the new scorecard here is based on the largest sample and on the latest nationally representative data. Third, the accuracy of the new scorecard compares well with that of others. And fourth, the scorecard here (or at least its predecessor based on the 2000 HIES, Schreiner, 2006a) is actually being used by local pro-poor organizations. Comparing poverty scorecards is not a mere academic exercise because many local, pro-poor organizations in Bangladesh already use very simple rule-of-thumb scorecards (such as a single indicator for land ownership or an index based on a handful of housing characteristics) for targeting and for measuring change over time. A simple, inexpensive scorecard with greater accuracy could help managers to improve their efforts to alleviate poverty in Bangladesh. 12

16 3.1 Grameen Bank The Grameen Bank probably the world s best-known microfinance organization (Dowla and Barua, 2006; Rutherford, 2006) designed its own poverty scorecard to measure the exit of its members from poverty through time. 4 The 13 indicators are: Characteristics of the residence: Is the roof made of tin or is the residence worth more than 25,000 Taka? Does the family use a sanitary latrine? Does drinking water come from a tube well, or has it been purified by boiling, pitcher filters, alum, bleach, or tablets? Do all children six and up go to primary school or have finished primary school? Ownership of assets: Do family members sleep on cots or beds? Do all family members have sufficient clothing for daily use? Do all family members have warm clothes for winter? Do all family members have mosquito nets? Status as a microfinance participant: Does the Grameen member pay a weekly installment of at least 200 Taka? Does the Grameen member have an average annual savings balance of at least 5,000 Taka? Does the family have diversified sources of income? Does the family eat three square meals per day throughout the year? Are all family members conscious about their health, with the ability to take immediate action and pay for medical expenses in the event of an illness? For Grameen s purposes, a household has exited poverty if it can answer Yes to all 13 indicators. Grameen s poverty scorecard is based on its in-house expertise and experience, and as such it is well-accepted by its staff. Some indicators, however, are subjective 4 Founded by Mohammad Yunus (winner of the 2006 Nobel Peace Prize), Grameen in March 2009 had about 8 million members (almost all rural women), about $0.7 billion in loans outstanding, and about $1 billion in deposit balances. Grameen inspired much of the worldwide microfinance movement as well as two other similar microfinance titans in Bangladesh, BRAC (Smillie, 2009) and ASA (Rutherford, 2009). 13

17 (such as Are all family members conscious of their health ) or unverifiable ( Does the family eat three square meals throughout the year? ). Furthermore, two indicators ( Weekly installment is at least 200 taka and Average savings is at least 5,000 taka ) are relevant only for microfinance participants. Unlike the scorecard in this paper, Grameen s scorecard is not benchmarked to an expenditure-based poverty line. While Grameen s definition of poverty is completely sensible, it is not quantifiable in the units typically used in poverty analysis. 5 Its accuracy is defined, not tested. Also, from the point of view of an expenditure-based poverty line, Grameen s scorecard it is too stringent; some households with per-capita expenditure above a given poverty line will not answer Yes to all 13 indicators. 3.2 Gwatkin et al. Gwatkin et al. (n.d.) apply to Bangladesh an approach used by USAID in 56 countries with Demographic and Health Surveys (Rutstein and Johnson, 2004). They use Principal Components Analysis to make a wealth index from simple, low-cost indicators available for the 10,500 households in Bangladesh s 2004 DHS. The index is like the poverty scorecard here except that, because it is based on a relative definition of poverty, its accuracy is unknown, and it can only be assumed to be a proxy for long- 5 Of course, this may be a great strength. 14

18 term wealth/economic status. 6 Other examples of the PCA-index approach are Stifel and Christiaensen (2007), Zeller et al. (2006), Sahn and Stifle (2003 and 2000), and Filmer and Pritchett (2001). The 20 indicators in Gwatkin et al. are similar in their simplicity and verifiability to those in the scorecard here: Characteristics of the residence: Presence of electricity Source of drinking water Type of fuel for cooking Type of toilet arrangement Type of floor Type of roof Type of walls Whether the household owns land Whether the household has a domestic worker not related to the head Ownership of consumer durables: Radio Television Telephone Bicycle Motorcycle or scooter Almirah (wardrobe) Table Chair or bench Watch or clock Cot or bed Sewing machine 6 Still, because their indicators are similar and because the flat max is important, carefully built PCA indices and expenditure-based poverty scorecards probably pick up the same underlying construct (such as permanent income, see Bollen, Glanville, and Stecklov, 2007), and they probably rank households much the same. Tests of how well PCA indices predict expenditure include Filmer and Scott (2008), Lindelow (2006), Wagstaff and Watanabe (2003), and Montgomery et al. (2000). 15

19 Gwatkin et al. has three basic goals for the PCA-based wealth index: Segment people by quintiles in order to see how health, population, and nutrition vary with socio-economic status Monitor (via exit surveys) how well health-service points reach the poor Measure coverage of services via small-scale local surveys These last two goals resemble the monitoring goals here, and the first goal of ranking households by quintiles is akin to targeting. As here, Gwatkin et al. present a ready-to-use index, although their format is more difficult because it has two pages, all points have 5 decimal places, no points are zero, and some points are negative. The central contrast between the scorecard here and the PCA index is the use/non-use of an absolute, expenditure-based poverty line. Thus, while both approaches can rank households, only the poverty scorecard can estimate quantitative, expenditure-based poverty status. Furthermore, relative accuracy (that is, ability to rank or target) is tested here more completely here than in Gwatkin et al.; generally, discussion of the accuracy of PCA indices rests on how well they correlate with health, education, or self-assessed poverty, even though their construction does not take any such correlation into account. 3.3 Wodon Wodon (1997) seeks indicators for targeting the poor. To this end, he develops a set of poverty scorecards based on expenditure in the 1991/2 HES (predecessor to the HIES). Targeting strength is tested via ROC curves (equivalent to the columns % of all households who are targeted and % of poor who are targeted in Figure 15 here). 16

20 Wodon compares scorecards with only housing indicators against broader scorecards and also against a series of one-indicator scorecards. He uses Logit (as does this paper) to construct all these scorecards for three areas (urban, rural, and Bangladesh as a whole) and for both national poverty lines. In all cases, Wodon estimates poverty likelihoods, but he does not report points. The five indicators in the housing scorecards are: Type of wall Type of roof Number and size of bedrooms Type of toilet arrangement Source of drinking water These are simple, inexpensive, and verifiable, but they are also likely to be highly correlated with each other; few houses with high-quality roofs have low-quality walls. The 13 indicators in the broader scorecards are: Household demographics: Number of babies (and its square) Number of children (and its square) Number of adults (and its square) Age of the male head/spouse (and its square) Age of the female head/spouse (and its square) Family structure Highest educational level attained by: Male head/spouse Female head/spouse Any other family member Main occupation of the household head Amount of land owned Religion Geographic location 17

21 Wodon calls this a determinants of poverty scorecard because the indicators are determined before current poverty status and so are not themselves caused by current poverty status. This scorecard turns out to target better than the others. In all Wodon s scorecards, the indicators are simple, inexpensive, and verifiable, although Wodon does not believe that they are feasible, saying (p. 2087) it is unlikely that we would have the necessary information to use the determinants of poverty model in practice. Even if we did, the implementation of a policy under such a complex set of indicators might be too difficult. In fact, Grameen s scorecard above is implemented and is much more complex, and BRAC and ASA are implementing the predecessor to the scorecard in this paper (Schreiner, 2006a). For measuring targeting accuracy, ROC curves are appropriate. Wodon s tests, however, are in-sample, meaning that they use the same data that was used to construct the scorecard. In-sample tests overstate accuracy, because all scorecards are overfit to some extent, meaning they capture not only universal, timeless relationships between indicators and poverty but also relationships that change through time or that appear in a particular sample solely due to chance. A better way to test scorecard accuracy (for targeting or for other purposes) is with out-of-sample tests that use data not used to construct the scorecard. This paper uses only out-of-sample tests. 18

22 3.4 Haslett and Jones Haslett and Jones (2004) use poverty mapping (Elbers, Lanjouw, and Lanjouw, 2003) to estimate poverty rates for Bangladesh at the lowest administrative rural unit (the union). They first construct a single poverty scorecard for Bangladesh as a whole using a single-stage, robust regression to estimate the logarithm of expenditure for the 7,440 households in the 2000 HIES, considering only indicators found also in the 2001 population census. The resulting poverty scorecard is then applied to the five-percent sample of the census data to estimate poverty rates for the lower and upper national lines for smaller areas than would be possible with only the 2000 HIES. Finally, Haslett and Jones make poverty maps that quickly show how estimated poverty rates vary across areas in a way that makes sense to lay people. The poverty mapping in Haslett and Jones has much in common with the poverty scoring here in that they both: Build scorecards with nationally representative survey data and then apply them to other data on groups that may not be nationally representative Use simple, verifiable indicators that are quick and inexpensive to collect Select indicators based on statistics, judgment, and experience to reduce overfitting Provide unbiased estimates Report standard errors for their estimates (or, equivalently, confidence intervals) Estimate poverty rates for groups Seek to be useful in practice and so aim to be understood by non-specialists Strengths of poverty mapping include that it: Has formally established theoretical properties Can be applied straightforwardly to measures of well-being beyond poverty rates Requires less data for construction and calibration Uses only indicators that appear in a census 19

23 Strengths of poverty scoring include that it: Is simpler in terms of both construction and application Tests accuracy empirically Associates poverty likelihoods with scores non-parametrically Estimates poverty likelihoods for individual households Reports simple formulas for standard errors and sample sizes The basic difference between the two approaches is that poverty mapping seeks to help governments design pro-poor policies, while poverty scoring seeks to help small, local pro-poor organizations to manage their outreach when implementing policies. 7 Haslett and Jones 21 indicators for Bangladesh are: Demographics: Household size Square of the difference between household size and the upazila mean Proportion of household members who are: Under five years of age Female Literate Dependency ratio (details not documented) Whether the household head has completed primary school Employment: Whether the main source of income is construction or transportation Proportion of household members who are: Employers Employees, family helpers, or other Self-employed Characteristics of the residence: Type of house Presence of electricity Type of toilet arrangement Source of drinking water 7 Another apparent difference is that the developers of poverty mapping say that it is inappropriate for targeting individual households or persons, while this paper supports such targeting as a legitimate, potentially useful application (Schreiner, 2008a). 20

24 Ownership of real estate: House Agricultural land Location: Urban/rural Division Census means at the level of the upazila: Household size Share of households with agriculture as the main source of income In addition, there are seven indicators that combine indicators. This complexity means that the scorecard cannot be used for on-the-spot targeting. Because the census does not measure expenditure, Haslett and Jones cannot test accuracy out-of-sample. They do report standard errors for estimated poverty rates, averaged across upazilas. For the lower (upper) national line, the 90-percent confidence interval for their scorecard s estimate of the poverty rate is +/ 6.4 (6.8) percentage points. Using the formula in Section 7 below and noting that information in Haslett and Jones suggests that the average number of households per upazila in the five-percent census sample was about 2,500, the 90-percent confidence interval for the standard error of the estimated poverty rate for the 2005 scorecard in this paper applied to a sample of n = 2,500 from the 2000 HIES is +/ 1.4 (1.5) percentage points for the upper (lower) national line. While the confidence interval for the scorecard here is about four times narrower than that in Haslett and Jones, the comparison is imperfect, both because all upazilas do not have 2,500 households in the census sample and because the figure here comes from a single nationally representative sample while Haslett and Jones figure is an 21

25 average across 507 upazilas, most of which are probably not nationally representative (Tarozzi, 2007; Tarozzi and Deaton, 2008). It would be better to consider both bias and standard error at the upazila level, but that is beyond the scope of this paper. 3.5 Kam et al. Like Haslett and Jones, Kam et al. (2004) use the five-percent sample of Bangladesh s January 2001 population census to make poverty maps, this time at the upazila level. They build their scorecard not with expenditure from the 2000 HIES but rather with income from a nationally representative 2000/1 survey of 1,888 households by the International Rice Research Institute. Kam et al. use two poverty lines based on the cost of 2,112 calories (or 1,800 calories) and 58 grams of protein derived from the consumption by rural households in the 2000 HIES, adding 40 percent for non-food purchases. Their scorecard is derived from ordinary least-squares regression on income with nine indicators: Education: Average years of schooling among working household members Number of adults who attended college Employment: Number of agricultural workers Number of non-agricultural workers Whether the household has a business Characteristics of the residence: Presence of electricity Quality of house Ownership of agricultural land Whether the household is Muslim 22

26 In addition, there are four more indicators that combine indicators. In general, the indicators in Kam et al. are simple, inexpensive, and verifiable. Overall, Kam et al. is less useful than Haslett and Jones. For example, a central strength of poverty mapping as developed by Elbers, Lanjouw, and Lanjouw (2003) is the reporting of standard errors, something Kam et al. do not do. 3.6 Zeller, Alcaraz V., and Johannsen Zeller, Alcaraz V., and Johannsen ( ZAJ, 2004) seek to help USAID microenterprise partners report on their participants poverty rates. To do this, they use ordinary least-squares regression to predict the logarithm of per-capita expenditure for 799 households from a nationally representative survey conducted specifically for ZAJ. Indicators are selected from a pool of about 700 candidates by an automated forward stepwise routine that maximizes R 2. The poverty line is $1.08/day 1993 PPP (Taka23.1/day), corresponding to a poverty rate in their sample of 36 percent. 8 ZAJ build a series of nine scorecards, progressively restricting the pool of candidate indicators to be simpler, less expensive, and more verifiable. For each scorecard, they test variants with 8, 13, and 18 indicators. For this paper, the most relevant scorecard is ZAJ s Model 7, as it considers only indicators rated as easily verifiable by the survey firm. 8 It is not reported whether this is a household-level or person-level rate. 23

27 The 13-indicator version uses: Household demographics: Household size (and its square) Age of the household head Whether the household head is a domestic worker Characteristics of the residence: Whether the house structure is good Whether there is an improved toilet Ownership of agricultural assets: Whether less than 50 decimals of land are owned, including homestead Value of milk cows owned Ownership of consumer durables: Value of radios, televisions, VCRs, and CD players Number of saris Number of mosquito nets Presence of blankets Geographic division Whether the household declares that it is not able to save Compared with indicators in the scorecard here, these are greater in number, more complex, more expensive, and less verifiable. In particular, it is not clear what is a good house, nor how to verify whether a household can save. Also, households may have trouble valuing their milk cows, radios, televisions, VCRs, and CD players. While ZAJ resembles this paper in that it seeks to estimate poverty rates for groups of households, it also differs in several ways. First, ZAJ do not discuss using its scorecards for targeting or for estimating changes in poverty rates for groups. Second, ZAJ do not report the points in its scorecards. Third, ZAJ s estimates are statistically biased, while those here are unbiased. 9 Fourth, ZAJ s measures of accuracy are 9 This follows from the fact that the indicator function ZAJ use to convert estimated expenditure into poor/non-poor poverty status is non-linear and discontinuous. 24

28 overstated because they based on in-sample tests (and their automated indicator selection only worsens overfitting). Fifth, ZAJ report no standard errors. Sixth, ZAJ s approach does not use poverty likelihoods but rather labels a household as 100 percent below or above a poverty line, even though some households with estimated expenditure on one side of a given line have true expenditure on the other side of the line. How does ZAJ compare with the scorecard here in terms of targeting accuracy? For Model 7 with 13 indicators and a poverty line of $1.08/day 1993 PPP (poverty rate of 36.0 percent) applied in-sample to its special-purpose 2004 survey, ZAJ report undercoverage of 49.8 percent (half of households with true expenditure below the line have estimated expenditure above the line) and leakage of 23.5 percent (one-fourth of households with true expenditure above the line have estimated expenditure below the line). For the scorecard here and the upper national line (poverty rate of 37.2 percent, Figure 2) applied out-of-sample to the validation sample from the 2005 HIES with a cut-off of 25 29, undercoverage of 45.0 corresponds to leakage of Thus, the scorecard here has less undercoverage and less leakage, and so better targeting. 3.7 IRIS Center IRIS Center (2007a) updates ZAJ and shares most of its strengths and weaknesses. After comparing several statistical techniques (and therefore increasing the risk of overfitting), IRIS selects a two-stage approach. In the first stage, a linear probability model (akin to the Logit here) identifies households with extremely high or 25

29 extremely low estimated poverty likelihoods. A second linear probability model is then applied to the remaining households, and those with an estimated poverty likelihood of less than 50 percent being counted as poor. This two-step approach was first used in poverty scoring by Grootaert and Braithwaite (1998), although it has been in the scoring literature for decades (see, for example, Myers and Forgy, 1963) and is a variant on the idea of boosting (Hand and Vinciotti, 2003; Friedman, 2001; Schapire, 2001). IRIS 38 indicators are: 10 Household demographics: Number of members Number of males Number of females Number of under the age of 14 and over the age of 60 Age of the household head Marital status of the household head Education: Highest class completed by the household head Number of members (excluding head) with no education Number of members (excluding head) whose highest class is primary school Employment: Whether the household head was a domestic worker in the past year Minimum wage acceptable to the main income-earning female member for eight hours of hard work during the post-harvest season Characteristics of the residence: Rooms Source of drinking water Whether a home improvement was made in the past three years Cost of any home improvements made in the past three years 10 IRIS does not report the actual scorecard, only the questionnaire used to collect data, so the actual indicators may differ from those listed here. 26

30 Ownership of agricultural assets: Number of milk cows Presence of a motor tiller Total value of irrigated agricultural land Whether the household had a very serious problem or failure in its animal production in the past three years Ownership of consumer durables: Number of radios Number of CD players Number of televisions Number of VCRs Number of ceiling fans Number of kantha (embroidered textiles) Number of saris Number of carts, wagons, or similar vehicles Area of homestead land Food security in the past year Social participation: Number of members in a trader s association Number of members in a cultural group Number of members in a political group Number of members in a school committee Number of first-degree relatives (mother, father, sister, brother) of the household head or spouse who got married in the past three years Whether any member has a withdrawable savings account of any type Location: Region Urban/rural Besides having almost four times as many indicators as the scorecard here, the IRIS indicators are also more complex, more expensive, and less verifiable. For example, an enumerator cannot verify responses that are concerned with events in the past (such as home improvements, problems with animal husbandry, food security, family marriages, or hypothetical reservation wages). Households also cannot easily supply the value of past home improvements or of their irrigated land. Finally, households may be unwilling to reveal whether they have a savings account. 27

31 IRIS preferred measure of accuracy is the Balanced Poverty Accuracy Criterion, and USAID adopted BPAC as its criterion for certifying poverty scorecards (IRIS Center, 2005). BPAC depends on the difference between the estimated poverty rate and its true value (a difference that is minimized by minimizing the absolute difference between undercoverage and leakage) and on inclusion, that is, the share of households who truly have per capita expenditure below a given poverty line and who are also correctly classified as below poverty line. The formula is: (Inclusion Undercoverage Leakage ) x [100 (Inclusion + Undercoverage)]. A higher BPAC implies more accuracy; for IRIS in-sample tests, BPAC is For the scorecard here and the upper national poverty line (the line that gives a poverty rate closest to that of the poverty line used by IRIS), out-of-sample BPAC is Analysis of poverty scorecards for Peru (Schreiner, 2009a) suggests that going from in-sample to out-of-sample can reduce BPAC by percent, or in the case of Bangladesh, from 72.1 down to 65.9 or Given the possibility of sampling variation on top of this, BPAC is probably about the same for IRIS as for the scorecard here. The main distinction between the scorecard here and IRIS is transparency. In particular, IRIS does not report: What data their scorecard is based on (although it appears to be the same as in ZAJ) The points associated with scorecard indicators Standard errors Whether poverty rates are at the household- or person-level 28

32 3.8 Cortez et al. Cortez et al. (2005) aim to improve the targeting of health services to individuals in Bangladesh. To this end, they construct a poverty scorecard with ordinary leastsquares on the logarithm of per-capita expenditure from the 2000 HIES. Their initial scorecard has about 40 indicators selected for their statistical significance. To get a more feasible tool, Cortez et al. winnow this initial scorecard down to 14 indicators: Number of household members Education: Number of members aged 16 and up who never attended school Highest educational attainment by any household member Characteristics of the residence: Presence of electricity Type of wall Type of roof Type of toilet arrangement Source of drinking water Ownership of consumer durables: Electric fans Televisions Dining-room furniture Drawing-room furniture Freezer Telephone (landline or mobile) As here, all indicators in Cortez et al. are simple, inexpensive, and verifiable. Furthermore, the points are simple, with only a single decimal place (compared the nodecimal-point scheme here). In general, Cortez et al. emphasize practicality, presenting a ready-to-use scorecard (as here), providing advice on implementation, and illustrating how to use the scorecard to estimate expenditure and to apply a targeting cut-off 29

33 defined at the 60 th percentile of expenditure (without adjustments for regional differences in cost-of-living). For all these similarities, there are also differences in that Cortez et al.: Do not report standard errors Estimate expenditure (not poverty likelihoods) Has 15 indicators (versus 10), two of which are continuous (versus none) Cortez scorecard uses the 2000 HIES, 11 and it can be reconstructed to enable comparisons with the scorecard here in terms of targeting accuracy and in terms of the bias and variance for estimated overall poverty rates. Schreiner (2006a) uses ROC curves (as in Wodon) to show that the predecessor to the scorecard here (based on the 2000 HIES) is better at targeting. For example, targeting the lowest-scoring 30 percent of households in an out-of-sample test with Cortez et al. targets 51.4 percent of the poor and 14.4 percent of the non-poor. The predecessor of the scorecard here is slightly better, targeting 55.3 percent of the poor and 10.4 percent of the non-poor. This advantage is maintained across all targeting cutoffs. To test the bias and precision of estimates of groups poverty rates at a point in time, 10,000 bootstrap samples were drawn out-of-sample. For Cortez et al., the difference between the estimated poverty rate and the true value has a mean of Cortez (p. 73) lists an indicator Has no private toilet. Whether toilet arrangements are private or shared, however, is not in the 2000 HIES. Based on the reported mean, the indicator must be Does the household use a temporary kacha latrine or open fields? 30

34 percentage points and a standard error of 1.2 percentage points. For the predecessor to the scorecard here, the mean difference is +0.5 percentage points with a standard error of 0.9 percentage points. Thus, the scorecard here is both more accurate (less bias) and more precise (less variance). 31

35 4. Scorecard construction About 100 potential indicators are initially prepared in the areas of: Family composition (such as household size) Education (such as school attendance of children) Housing (such as the main construction material of the walls) Ownership of durable goods (such as televisions and wristwatches) Employment (such as whether any household member works for a daily wage) Each indicator is first screened with the entropy-based uncertainty coefficient (Goodman and Kruskal, 1979) that measures how well the indicator predicts poverty on its own. Figure 4 lists the candidate indicators, ranked by uncertainty coefficient. Responses for each indicator in Figure 4 are ordered starting with those most strongly associated with poverty. The scorecard also aims to measure changes in poverty through time. This means that, when selecting indicators and holding other considerations constant, preference is given to more sensitive indicators. For example, ownership of a television is probably more likely to change in response to changes in poverty than is the education of household members. The scorecard itself is built using the $1.25/day 2005 PPP poverty line and Logit regression on the construction sub-sample (Figure 2). Indicator selection uses both judgment and statistics (forward stepwise, based on c ). The first step is to use Logit to build one scorecard for each candidate indicator. Each scorecard s accuracy is taken as c, a measure of ability to rank by poverty status (SAS Institute Inc., 2004). 32

36 One of these one-indicator scorecards is then selected based on several factors (Schreiner et al., 2004; Zeller, 2004), including improvement in accuracy, likelihood of acceptance by users (determined by simplicity, cost of collection, and face validity in terms of experience, theory, and common sense), sensitivity to changes in poverty status, variety among indicators, and verifiability. A series of two-indicator scorecards are then built, each based on the oneindicator scorecard selected from the first step, with a second candidate indicator added. The best two-indicator scorecard is then selected, again based on c and judgment. These steps are repeated until the scorecard has 10 indicators. The final step is to transform the Logit coefficients into non-negative integers such that total scores range from 0 (most likely below a poverty line) to 100 (least likely below a poverty line). This algorithm is the Logit analogue to the familiar R 2 -based stepwise with leastsquares regression. It differs from naïve stepwise in that the criteria for selecting indicators include not only statistical accuracy but also judgment and non-statistical factors. The use of non-statistical criteria can improve robustness through time and helps ensure that indicators are simple and make sense to users. The single poverty scorecard here applies to all of Bangladesh. Evidence from India and Mexico (Schreiner, 2006b and 2005a), Sri Lanka (Narayan and Yoshida, 2005), and Jamaica (Grosh and Baker, 1995) suggests that segmenting scorecards by urban/rural does not improve accuracy much. 33