Data quality analysis of the NRVA 2007/08 Beatriz Godoy 1, consultant July-August, 2009

Transcription

1 Data quality analysis of the NRVA 2007/08 Beatriz Godoy 1, consultant July-August, 2009 The NRVA 2007/08 data set is a nationally representative, multi-topic household survey data for Afghanistan. It covers various topics of living standard and activities of households in Afghanistan such as agriculture, labor, migration, gender, etc. The data were collected from September 2007 to September The NRVA data have been entered, cleaned, and processed and are currently ready for analysis. The purpose of this assignment is to assess the quality of the NRVA data, from an independent view. EXECUTIVE SUMMARY To carry out a survey such as this one, under normal conditions involves a considerable challenge, and conducting it in Afghanistan was even still greater on account of the conditions which the country is currently undergoing. This survey was successfully conducted over a span of twelve months, despite the usual drawbacks in this type of operations, -instructions that were clearly spelt out initially but that are forgotten by the enumerators later, or questions that were misunderstood by respondents, and the like- all of which were solved in an effortless manner. Within the frame of this consultancy I analyzed different indicators of the data quality, and I can say based on our experience in other countries that have carried out similar surveys that this one is among the best ones. The results obtained do not show significant errors; much to the contrary, the few errors indicate a strong supervision of field teams, since experience has shown over and over again that these particular surveys are very sensitive to mistakes from the interviewer's side. I have developed a set of indicators in the body of this report 2, which is divided into the following six chapters: (1) completeness of household s data, (2) completeness of individual s data, (3) demographic consistency, (4) empty records and duplicates, (5) internal consistency of sections and (6) multidimensional quality controls. The result of assessing the quality of the NRVA 2007/08 data gives excellent indicators; consequently the users are advised to begin the analytical exploitation of this rich database. 1 Address: Sistemas Integrales Casilla Santiago, Chile. Phones: (56-2) and (56-2) Fax: (56-2) beatriz.godoy@ariel.cl 2 Note: The list of possible quality indicators for a survey like the NRVA 2007/08 has almost not limit, therefore I have selected the ones that according to our experience are more sensitive to errors.

2 1.- COMPLETENESS OF HOUSEHOLD S DATA In this stage of the quality analysis I revise the completeness of household s data in terms of the completed sections or chapters of the questionnaire. There are sections that are obligatory, that is to say, that they must be available at every household, and others that must be completed depending on its demographic structure. Among the obligatory sections the following must be mentioned: 1: Household Roster, 2: Housing, 5: Assets and Credit, 12: Household Expenditures, 15: Food consumption last 7 days and 16: Iodized salt, avian flu and household expenses. There are others which may not be applicable to a household such as for example 3: Livestock, 4: Agriculture might be empty if the household has no farming activities or 8: Sources of household income, 10: Cash for work, Food for work could be blank if the household subsists due to pensions or other benefits; but for each of these sections there is a filter question at the beginning; and at the least, this question must be answered (see example in figure 1). Figure 1: Example of filter question In the case of some specialized sections, the complexities used at this stage consist in checking the number of individuals who meet the application criteria of the section corresponding to the number of people that responded it. These criteria will invariably be found in section: 1: Household Roster. Section 6: Education (for household members 6 years old and more) Section 9-A: Labor for members 6-15 yrs. Section 9-B: Labor and Migration, for members 16 years and over. Section 17: Number of children born and marriage information, for every married woman 49 years and less. Section 19: Immunization and child health, for children less than 5 years. Section 20: Women's activities, for women 10 years and above. The result of this exploration is that very few households have incomplete data for some section(s), see Figure 2:

3 Figure 2: Completeness of household data Total number of Households: 20,576 Individual sections Consistent records Number of households with missing information on the section Number Percent Number Percent Cover 20, % 0.00% Section 1: Household Roster 20, % 0.00% Section 2a: Housing and Utilities 20, % 0.00% Section 2b: Housing and Utilities 20, % % Section 3: Livestock 20, % % Section 4: Agriculture 20, % % Section 5: Assets and Credit 20, % 0.00% Section 6: Education (for househiold members 6 years old and more) 20, % 0.00% Section 7: Disabilities (for all household members) 20, % 0.00% Section 8: Sources of HH income 20, % % Section 9a: Labor for members 6-15 yrs 20, % % Section 9b: Labor and migration 16 and over 20, % % Section 10: Cash for work, Food for work 20, % % Section 11a: Migration and remittances 20, % % Section 11b:Migration and remittances 20, % % Section 12: Household Expenditures 20, % % Section 13: Household shocks and coping strategies 20, % % Section 14: Finale male interview 20, % % Section 15a: Food consumption las 7 days 20, % % Section 15b: Food consumption las 7 days 20, % % Section 16: Iodized salt, avian flu and hh expenses 20, % % Section 17: Number of children born and marriage information (for ever married women up to 49 years) 20, % % Section 19: Immunization and child health (for all children under 5) 20, % 0.00% Section 20: Women's activities (for all females in the household, ages 10 and older) 20, % % Section 20: Women's activities (for all females in the household, ages 10 and older) 20, % 0.00% The results are excellent; nevertheless I recommend to see in detail the sections where information is missing considering the objective of the analysis so as to identify the degree of seriousness if any at all that this lack of data might involve.

4 2.- COMPLETENESS OF INDIVIDUAL S DATA In this chapter I analyze the completeness of the data at the individual level. The checks are based on the following: all household members 6 years and more should answer Section 7: Education, all household members should answer Section 7: Disabilities, all household members aged 6-15 must answer Section 9- A: Labor, all household members aged 16 years and more must answer Section 9-B: Labor and Migration, all women ever married that are 49 years or less must answer Section 17: Number of children born and marriage information, all children less than 5 years must answer Section 19: Immunization and child health, and all women 10 years and above must answer Section 20: Women's activities. Total number of individuals in Section 1, Household Roster: Figure 3: Completeness of individual sections 152,284 Individual sections Consistent records matches the criteria but are not present in the section Nb of household members that: fail the criteria but are present in the section don't have complete information on section 1 (Roster) are not in section 1 (Roster) Number Percent Number Percent Number Percent Number Percent Number Percent Section 1: Household Roster 152, % 0.00% 0.00% % 0.00% Section 6: Education (for househiold members 6 years old and more) 152, % 0.00% % 0.00% 0.00% Section 7: Disabilities (for all household members) 151, % % 0.00% 0.00% 0.00% Section 9a: Labor for members 6-15 yrs 152, % 0.00% % % 0.00% Section 9b: Labor and migration 16 and over 152, % 0.00% % % 0.00% Section 17: Number of children born and marriage 152, % 0.00% % 0.00% 0.00% information (for ever married women up to 49 years) Section 18: Recent births (children born since August 2005) 151, % 0.00% % % % Section 19: Immunization and child health (for all children under 5) Section 20: Women's activities (for all females in the household, ages 10 and older) 151, % 0.00% % % 0.00% 151, % % % % 0.00% In a country such as Afghanistan, it is a challenge to have obtained percent of women in section 20, therefore I don t see this 891 of missing woman as indication of quality of the data sets. In some countries the issue of disability is culturally very badly seen, this could be the reason why some individuals didn t answer Section 7. Anyway, this figure of 475 represents only 0.31 percent of the individuals. 3.- DEMOGRAPHIC CONSISTENCY In this chapter I devote the efforts exclusively to analyze the demographic consistency of the households. The indicators used for this purpose are: (1) Each household must have a head, and only one. (2) Each Head must be fifteen years old or more. (3) If a household member has a spouse, and there is a spouse ID CODE in question 5, then this person should be in the household roster. (4) If a household member has a spouse, this spouse should have different sex. (5) If a household member has the father living in the household, then the ID CODE registered in question 7 must be in the roster (6) ID CODE of father should correspond to a MAN (7) The age difference between the individual and the father should be more than 15 years (8) If a household member has the mother living in the household, then the ID CODE registered in question 9 must be in the roster (9) ID CODE of father should correspond to a WOMAN (10) The age difference between the individual and the mother should be more than 12 and 55 years.

5 Figure 4: Demographic consistency of households 1.- Number of household's head per household 6.- Sex of father n percent n percent One head 20, male 94, Two head female Head missing Total 95, Total 20, Age difference between father and household member 2.- Consistency of age of the household head n percent n percent less than 15 years less than 15 yrs old years or more 151, >15 years old 20, Total 152, Total 20, Existence of ID CODE of mother 3.- Existence of ID CODE of spouse n percent n percent mother not found in roster spouse not found in roster mother found in roster 151, spouse found in roster 152, Total 152, Total 152, Sex of mother 4.- Consistency of spouse's sex n percent n percent male Same sex female 100, Different sex 50, Total 100, Total 50, Age difference between mother and household member 5.- Existence of ID CODE of father n percent n percent between 12 and 55 years 151, father not found in roster less than 12 years father found in roster 151, more than 55 years Total 152, Total 152, As shown in Figure 4, the results do not show significant errors; on the contrary, the few errors indicate a strong supervision of the field teams, since experience demonstrates that this chapter of the questionnaire is very sensitive to errors from the interviewer's side.

6 4.- EMPTY RECORDS AND DUPLICATES Prior to performing an in depth analysis, it is advisable to identify any empty or duplicated records. By empty records I understand those records which have no data in the relevant variables of the subject registered in the file. In this set of data files, and prior to this revision, the variables cid, stratum, hh_weight (and at times mem_weight ) were added to all the data files; therefore these variables do not count when looking for empty records. There are very few cases (see Figure 4 below), and the analysts should decide whether to they should be dropped from the original files or not. Figure 5: Number of empty records empty records file name name of the section total number of records n percent s_m.dta Cover male 20, s_f.dta Cover female 20, s_1.dta Section 1: Household Roster 152, a.dta Section 2a: Housing and Utilities 20, b.dta Section 2b: Housing and Utilities 20, dta Section 3: Livestock 20, dta Section 4: Agriculture 20, a.dta Section 5: Assets and Credit (question 1) 20, b.dta Section 5: Assets and Credit (questions 2-5) 71, c.dta Section 5: Assets and Credit (questions 6-27) 20, dta Section 6: Education (for househiold members 6 years old and more) 152, dta Section 7: Disabilities (for all household members) 152, dta Section 8: Sources of HH income 20, a.dta Section 9a: Labor for members 6-15 yrs 48, b.dta Section 9b: Labor and migration 16 and over 73, dta Section 10: Cash for work, Food for work 20, a.dta Section 11a: Migration and remittances 21, b.dta Section 11b:Migration and remittances 20, dta Section 12: Household Expenditures 20, dta Section 13: Household shocks and coping strategies 20, dta Section 14: Finale male interview 20, a.dta Section 15a: Food consumption las 7 days 20, b.dta Section 15b: Food consumption las 7 days 1,869, dta Section 16: Iodized salt, avian flu and hh expenses 20, dta Section 17: Number of children born and marriage information (for ever married women up to 49 years) 22, dta Section 18: Recent births (children born since August 2005) 12, a.dta Section 19: Immunization and child health - for all children under 5 (questions 1-28) 24, b.dta Section 19: Immunization and child health - for all children under 5 (questions 29 and 30) 20, a.dta Section 20: Women's activities - for all females in the household, ages 10 and older (questions 1-10) 20, b.dta Section 20: Women's activities - for all females in the household, ages 10 and older (questions 11-44) 47, As for duplicates, there are identical records only in files 5b (Section 5: Assets and Credit - questions 2-5) and 11b (Section 11b: Migration and remittances). But section 5b was designed so that it is very possible that more than one item with the same characteristics are recorded for the same household. And under Section 11b, it is also possible that the same household receives remittances from more than one person with exactly the same characteristics (relationship to head, sex, place of residence and occupation). Therefore I conclude that there are no true duplicates to be dropped from the database. 5.- GENERAL CONSISTENCY OF SECTIONS Section 2: Housing and utilities. I checked for out-of-range values for all qualitative variables, for internal skip errors and there is almost no error. Section 3: Livestock. (1) The number of animals owned in question 3.1 and the number of productive females in question 3.3. There are only 2 households with an inconsistency in this regard. (2) The ID CODES registered in question 3.5 must exist in Section 1: Household Roster. There are 147 households (0.7 percent) with at least one of the ID CODES on that question that are not in Section 1. Section 4: Agriculture. I checked that the quantities of land are consistent: (3) The area cultivated during the most recent summer cultivation season must be less or equal to the

7 total area of land the household members had access to: {Q_4_14 <= Q_4_4 + Q_4_6 + Q_4_7 + Q_4_8}. There is only one household failing this check. (4) The area sharecropped-out, rented-out, mortgage-out, or left fallow during the most recent summer cultivation season, must be less than or equal to the total area of land owned by the household members: {Q_4_4 >= Q_4_9 + Q_4_10 + Q_4_11 + Q_4_12}. There are fifty-nine household failing this check (representing 0.29 percent of the households). (5) The area of rain-fed land cultivated during the most recent summer cultivation season must be less than or equal to the total area of rain-fed land the household members had access to: {Q_4_27 <= Q_4_22 + Q_4_24}. There are four households failing this check. (6) The area of rain-fed land left fallow in the most recent summer cultivation season must be less than or equal to the total area owned and sharecropped-in: {Q_4_25 <= Q_4_22 + Q_4_24}. No household fails this check. (7) The area of garden plot the household had access to in the most recent summer cultivation season must be greater than or equal the detail given in question 4.31b: {Q_4_31a >= Q_4_31b_1 + Q_4_31b_2 + Q_4_31b_3 + Q_4_31b_4}. Fifteen households fail this check. Section 5: Assets and Credits. Question 5.1 How many of the following items does your household own? against the items listed in questions 5.2 to 5.5; see figure below. Figure 6: Section 5, Assets and credits I checked that all the items owned by the household (in question 5.1) are listed in questions 5.2 to 5.5. There are less than 3% of the households that present an inconsistency between the items reported in 5.1 and the detail given in questions 5.2 to 5.5. Section 6: Education. The age should be consistent with the highest level and year attended by the household members. I can only check that the age is not less than a minimum for each level and year, assuming that the minimum age to attend primary grade 1 is 5 years. See results below in Figure 7:

8 Figure 7: Age vs. Highest level of education and year attended n percent Too young for this level and year Consistent 41, Not checked 109, Total 152, Section 8: Sources of household income. There are two sources of income that I can check against other parts of the questionnaire: (1) Section 3: Livestock and the following sources of income: 2 Livestock production for home consumption 7 Prod & sales of livestock & products If any of the sources above appear in Section 8, then the answer to question 3.1 must be 1:YES. There are 74 (0.4 percent) households that fail this check. (2) Section 4: Agriculture and the following sources of income: 1 Crop production for home consumption 3 Production & sale of field crops 4 Prod & sales of cash crops (except Opium) 5 Production & sale of opium 6 Prod & sales of orchard products If any of the sources above appear in Section 8, then the answer to question 4.1 must be different than 4:NO. There are 136 (0.7 percent) households that fail this check. Section 12: Household expenditures. The fact that a household has information on this section does not necessarily imply that the information is usable. I computed the total amount of money spent in all nonfood items per household, during the past 30 days. There are 34 households (0.17 percent) that report zero expenditure in non-food items during the past 30 days. This is indeed not an error, because it could very well be that a household did not buy any. Section 15b: Food consumption in the past 7 days. I also computed the total quantity consumed by the household members (as an absolute number), and there are 18 households that have zero quantity of food consumed during the past 7 days, but one of them reported to have eaten several meals outside the household (question 15.3). Subsequently there are 17 households (0.08 percent) without food consumption at all during the past 7 days. 6.- MULTIDIMENSIONAL QUALITY CONTROLS DAILY CALORIE PER CAPITA INTAKE CONTROL This control uses the daily per capita energy provided by each food item in Section 15: Food consumption in the last 7 days, to perform various tests on the likelihood of the reported data. The purpose of these tests is to detect probable reporting or data entry errors and not to assess nutritional adequacy at this stage. This is why I do not use adult equivalents; on the contrary, I assume that all household members are similar and they should consume roughly more than 800 kcal per capita per day and less than 4,000 kcal per capita per day. The formula used is as follows: { Energy provided by a food item (Kcal/person /day) } = { Food intake (in unit of measurement of the food item) } x { Energy value of one measurement unit of food (kcal) } / { Number of person days }

9 Out of the 20,576 households, 32 do not have information on this section at all, and another 18 have zero food consumption. For the remaining households I proceeded as follows: 1. I computed the total energy provided by each food item. The energy value of 1 unit of food is taken from the food composition table shown in Annex I. 2. I compute the number of person days as 7 times the household size. I correct this value using the answer to question 15.2 How many meals were eaten by guests from the household cooking pot in the last 7 days?. If I assume that a person eats, on average, three times a day, then I add { 7 * (q_15_2 / 3) } to get the corrected Number of person days. I found that 2.8 percent of the households report a daily per capita consumption of less than 800 calories and that 3.7 percent of households consumed more than 4,000 calories per day and per capita. CASH BALANCE CONTROL The purpose of this control is only to detect values that are suspicious of being anomalous (outliers) in the variables related to the cash expenses and cash income. With this clear and unique objective I constructed two variables: total yearly income in cash of the households and total yearly expense in cash of the households. Next, I calculated the difference in the logarithms of these two variables and I searched for extreme values. The total yearly income in cash of the households includes the following questions in the questionnaire: Income from livestock activities, question 3.15 Largest loan obtained in cash, question 5.19 (if question 5.15 = cash) Annual income from question 8.4 The sum of question 9.20 for all members of the household. And the total yearly expense of the households considers: Rent in question 2.12 or 2.46 Water expenses in questions 2.33 and 2.38 (main and secondary source) Amount paid for the items acquired during the past 12 months, in question 5.4 Household expenses from section 12 Women s expenses in and I found only about 0.6 percent of the households that could be examined in greater detail because some of the values seem too high or too low, Figure 8 is a scatter plot showing the relationship between the two variables so computed using logarithm scales.

10 Figure 8: Cash income and expenditures Income Expenditure I haven t imputed a value to the amount spent on the food consumed by the household members in one year; therefore it is very likely that income looks higher than expenditures. Anyhow the cash income and expenditures are never expected to perfectly match in these surveys. However, the points too far from the diagonal probably reveal recording or under reporting errors

11 Annex I Food composition table item name unit kcal item name unit kcal 01 rice Kg 3, radish Kg wheat flour Kg 3, turnip Kg purchased nan piece cabbage Kg barley Kg 3, leek Kg maize corn broccoli Kg beans Kg 3, hot pepper Kg mung Kg 3, wild leaves Kg chickpeas Kg 3, coriander Kg lentils Kg 3, mint Kg macaroni Kg 3, dried tomato Kg 2, other bread Kg 56 dried vegetable Kg 2, beef Kg 1, pickled vegetable Kg 1, veal Kg 1, green bean Kg lamb Kg 2, other vegetable Kg 15 goat Kg 1, apple Kg chicken Kg 1, grapes Kg liver Kg 1, melon Kg dried meat Kg 6, peach Kg fish Kg dried apricot Kg 2, other meat Kg 65 orange Kg milk Kg plum Kg milk powdered Kg 5, pomegranate Kg yogurt Kg 1, pear Kg Curd Kg banana Kg krut Kg 4, raisins Kg 2, dogh Kg fresh mulberries Kg ghee Kg 8, dried mulberries Kg 3, butter Kg 6, mangoes Kg cheese Kg 3, walnut Kg 2, eggs Nb pistachio Kg 3, other dairy Kg 76 almonds Kg 2, vegetable oil Kg 8, other fruit Kg 33 animal fat Kg 9, white sugar Kg 3, other oil Kg 79 brown sugar Kg 3, potato Kg honey Kg 3, sweet potato Kg chocolate Kg 3, onion Kg black tea Kg 0 38 tomato Kg green tea Kg 0 39 okra Kg bottled water Lt 0 40 spinach Kg other beverage Lt 41 cauliflower Kg salt Kg 0 42 eggplant Kg black pepper Kg 2, carrots Kg ginger garlic Kg 1, pumpkin Kg tomato sauce Kg cucumber Kg mixed spices Kg 3, other spices Kg