Data quality assessment of the household budget survey Year 2008 July 2010 1
I. Introduction On evaluating the quality of the results of statistics, the goal is to achieve two fundamental objectives: To detect the errors that have been produced during the different stages of their compilation. To provide users with detailed information regarding the quality of the data that they deal with. The detection of the errors produced should not be reduced to a mere numerical presentation of them. The primordial objective should be their analysis, in order to decipher the possible causes leading to them. This is important, even essential, in all statistics, so as to improve the quality thereof. Continuous surveys such as the Household Budget Survey (HBS) also include the attraction of the immediate collection of their results, avoiding in parallel the deterioration of the quality of all of the routine work that this type of survey entails. To reduce the errors unrelated to sampling allows us to improve the quality of the estimates, for the purpose of obtaining acceptable levels of error, and maintaining them over time, which allows for a more adequate study of the resulting time series. On valuing the results of an assessment program, it is necessary to bear in mind the conditioning factors under which the surveys are conducted, which prevent, in many cases, evading the errors later detected in the assessment, with the compilers of the statistics still being conscious of the possibility of their presence. However, the supply of the information on the limitations of the data is an unavoidable duty, since an inappropriate use of the figures can cause the failure of socioeconomic and demographic plans and projects, and falsify conclusions on measures developed by politicians, economists and the remaining users of the statistics. The current volume publishes the data relating to the quality of the HBS in the year 2008. II. Quality of the data The errors that affect the entire survey can be grouped into two large classes: Errors due to sampling, caused by obtaining data via samples. Nonsampling errors, which are common in all statistical research, whether the data is obtained by sampling or by census. Chronologically, the first objective of those statisticians interested in the subject, both from a theoretical point of view and from the perspective of application, has been the calculation of the sampling error of the estimators. The importance of the sampling error calculation methods resides in the following fact: knowledge thereof enables, on the one hand, limitation within the confines of a confidence interval, the real value of an estimated parameter, and on the other, quantification of design efficiency as per the aforementioned parameter; 2
moreover, its analysis enables the statistician to choose the most efficient design from a series of alternatives, taking into account the resources available. The natural indicator of the accuracy of an unbiased estimator is its standard deviation, since with a given design, an unbiased estimator is more accurate, the more possible estimates are concentrated around the actual value. Accuracy increases with sample size, although design features also influence this: stratification, hierarchy of sampling units, selection method, etc., and the nature of the variables studied. The size of the sample is limited by the resources; the design is limited by the availability of basic structural information; and the nature of the variables is an element which cannot be acted upon. The errors other than sampling errors may occur in any of the phases of the statistical process: before collecting the data, during the information collection and in the operations subsequent to collection, it being possible to group data as actual fieldwork errors and otherwise. We may include among the former, among others, errors in collecting information, whether due to deficiencies on the part of agents or on the part of unsuitable informants, incorrect statements or nonresponse. Included in the latter are framework deficiencies, inadequacies in definitions and questionnaires, encoding or recording errors, etc. The study and application of statistical methods for assessing errors other than sampling errors, and the subsequent measuring of their influence on the end results, is more recent than that relating to sampling errors. One of the procedures followed in order to assess data quality, and which is applied in the HBS, consists of repeating the interview, shortly after having carried out the original interview, with part of the surveyed units. Through the comparison of the data collected in both interviews for the same units, it is possible to estimate the quality of the results, and provide the users with some numerical indices regarding said quality. This procedure is based on the model by Hansen, Hurwitz and Bershad, applied by the United States Census Office. In this report, only errors other than sampling errors are analysed; errors due to sampling are published with the survey data. III. General considerations regarding the survey To conduct the survey in 2008, an annual sample of 2,470 census sections has been selected (the 2,392 already selected in the year 2006, together with 78 additional sections in Comunidad Foral de Navarra, due to a partnership agreement with the Statistics Institute of Navarra, doubling the sample in that Autonomous Community), distributed throughout the national territory, visiting in each one of them ten randomlyselected dwellings. In each section, there is a listing of ten reserve dwellings that shall be used, as necessary, to make any required substitutions. The sample sections (and therefore, also the dwellings selected therein) are grouped into two rotation shifts, with half of the sections thereof corresponding to each one of them. Each year, the dwellings corresponding to a rotation shift are renewed (those from shift 1 one year, and those from shift 2 the following year), in such a way that the dwellings selected collaborate during two 3
consecutive years, after which they are replaced by other dwellings from the same section. In the year 2006, as it coincides with the beginning of the survey, all dwellings are in the first actual interview, irregardless of the rotation shift to which they belong. In order to enable the implementation of the survey, in the year 2007, dwellings from rotation shift 1 were replaced, despite having only collaborated in the survey for one year. In 2008, the dwellings from rotation shift 2 have been replaced, and in 2009 those from rotation shift shall be replaced again, and so on and so forth. When a reserve dwelling replaces an original dwelling, it acquires the same rotation shift as the original dwelling, and therefore, will be replaced by another when necessary (renewal of the sample) in accordance to it, even if it has not completed the two years of collaboration. In each dwelling, the household(s) residing therein is/are interviewed. The annual collaboration of each household takes place over the course of a twoweek period, in which all types of expenditure are requested by direct notation (household Book of Accounts) throughout those fourteen days, as well as the individual expenditure of each member in the first week of the twoweek period (individual Books of Accounts). The remaining information (Household file, paying of bills and other monthly, quarterly and annual expenditure) is requested by interview over the course of the twoweek period. IV. Nonresponse in the selected dwellings Within the dwellings selected for the sample, for part of them it is not possible to obtain information, either because they do not form part of the group being studied, due to not being used as a permanent family residence, or due to different reasons (refusal, absence, etc.) it is not possible to obtain data from the households resident therein. These situations, which the interviewer may encounter on carrying out her/his work, receive the name of incidences, and are described below. IV.1 Incidences concerning dwellings The selected dwellings are classified, according to the situation they are in at the time of the interview, as: Surveyable dwellings: those dwellings that are used for all or most of the year as a regular residence. Unsurveyable dwellings, can be: Empty: those dwellings that are unoccupied for all or most of the year, due to being empty, in ruins or seasonal. Unlocatable: those dwellings that cannot be located on the land with the address that appears in the work order. 4
Intended for other purposes: those premises intended in their entirety for purposes other than those of a family residence (for example, commercial premises, storage, offices, etc). Dwellings selected previously: those dwellings that, having been selected previously (less than five years ago) in the sample of the Household Budget Survey or any other population survey, and having collaborated therein, are selected again. Unavailable dwellings: those dwellings that cannot be accessed to conduct the interview, generally due to adverse climatological circumstances (snowstorms, floods, etc.) or due to the absence of adequate roads to access them. IV.2. Incidences concerning households In the dwellings that are surveyable, all of the households residing therein (there may be one, which is the most common, or more than one) are studied. In the households that reside in the surveyable dwellings, the following situations may occur: Surveyed: when the household collaborates in the survey. The collaboration may be total or partial, depending on the amount of information that the household provides. Refusal: when the household refuses to collaborate in the survey. Absence: when the interviewer does not find any member of the household in the subsequent visits made to the dwelling. Inability to respond: when all members of the household are incapacitated to collaborate in the survey, due to illness, disability, lack of knowledge of the language, etc. Refusals and absences may take place at any time throughout the collaboration period of the household; inabilities to respond, however, logically are detected at the time of the first contact with the household. The set of all refusals, absences and inabilities to respond constitute what is known as the nonresponse of the survey. IV.3. Treatment of the incidences A. Incidences concerning dwellings If the dwelling is surveyable, the household is studied. Empty and unlocatable dwellings and those intended for other purposes are replaced by reserve dwellings, and unavailable dwellings receive the same treatment as absent households. In the case of the dwellings that were previously selected in another population survey, when this situation is detected prior to beginning the field work, the 5
dwelling shall be replaced by the first available valid reserve dwelling, without it needing to be visited, assigning it the PS incidence (previously selected). In the event that the previous collaboration was not detected prior to beginning the fieldwork, but rather during the visit to the dwelling, there are two possible treatments: If the human group that resides in the dwelling accepts collaborating in the survey, it is interviewed normally, considering, in this case, the dwelling to be surveyable, and the household to be surveyed. If the human group does not accept collaborating due to a prior collaboration, the dwelling is replaced by the first available valid reserve dwelling, assigning it the PS incidence. B. Incidences concerning households Surveyed: the household is interviewed. Refusal: depending on the moment when the refusal takes place, the treatment will be different, with three possible situations: the household may be replaced, it may be a partial collaborator, or there may be a loss of the sample. This last case will occur when the refusal takes place at a time when a replacement is no longer feasible, and so long as, until that time, not enough information has been collected to consider it a partial collaborator. Absence: the dwelling is visited again as many times as possible, and if it is not possible to contact anyone, before reaching a loss of the sample, it is replaced. When it is ascertained that the absence will be definitive, the dwelling is replaced, even if it is the first visit. In cases of absences in replacement households, it might occur that there is a loss of the sample. Inability to respond: the dwelling is replaced by the first available valid reserve dwelling. IV.4. Failure to update the framework As commented previously, a dwelling is defined as unsurveyable in the HBS when, at the time of the interview, it is empty, it is a seasonal dwelling, it is intended for other purposes, or it is unlocatable at the address that appears in the selection listing. These cases are indicative of the survey framework not being updated or having errors, and these units may be considered erroneous inclusions in the framework. When they are detected on going to conduct the interview, they are never included in the survey, being replaced by other surveyable dwellings, as mentioned above, and therefore, there is no decrease in the size of the sample, unless it is impossible to carry out the replacement. Table 1 presents the distribution of the incidences in the theoretical sample (selected original dwellings), offering therein the breakdown of the information according to the number of the interview (first or second) and the type of municipality (provincial capital or remaining municipalities). 6
For the correct comprehension of the data from table 1, it is convenient to clarify two matters: first of all, due to the households only being those studied in the surveyable dwellings, the table data corresponds in part to dwellings (data corresponding to the unsurveyable dwellings, dwellings previously selected and unavailable dwellings) and in part to households (the data corresponding to surveyed, refusals, absences and inabilities to respond, encompassed in the surveyable section); secondly, the percentages of the incidences in dwellings have been calculated with regard to the total selected, whereas the percentages corresponding to the incidences in households have been calculated as compared with the total surveyable, this being the reason why two 100% appear in each column. It may be observed that, firstly, the incidences with less weight on a global level of the sample are, by this order, the dwellings previously selected, the unavailable dwellings, those intended for other purposes and those that are unlocatable, whose percentages are below 1 percent of the selected dwellings. The percentage of dwellings with an inability to respond slightly surpasses 1 percent of the surveyable dwellings, by which it also may be said that they have little importance from a quantitative point of view. Due to this, no more commentaries shall be made below regarding these incidences, on considering them of little interest. 7
1. Distribution of incidences in the theoretical sample Dwellings / households according Total First interview Second interview to type of incidence No. % No. % No. % Total Selected 23,869 100.00 11,915 100.00 11,954 100.00 Previously selected 33 0.14 25 0.21 Unavailable 40 0.17 29 0.24 11 0.09 Unsurveyable 1,571 6.58 1,172 9.84 399 3.34 Empty 1,309 5.48 956 8.02 353 2.95 Intended for other purposes 76 0.32 61 0.51 15 0.13 Unlocatable 186 0.78 155 1.30 31 0.26 Surveyable 22,225 100.00 10,689 100.00 11,536 100.00 Surveyed 15,484 69.67 6,376 59.65 9,108 78.95 Refusals 3,514 15.81 2,220 20.77 1,294 11.22 Absences 2,982 13.42 1,925 18.01 1,057 9.16 Inability to respond 245 1.10 168 1.57 77 0.67 Capitals Selected 8,237 100.00 4,135 100.00 4,102 100.00 Previously selected 6 0.07 6 0.15 Unavailable 9 0.11 8 0.19 1 0.02 Unsurveyable 384 4.66 263 6.36 121 2.95 Empty 303 3.68 197 4.76 106 2.58 Intended for other purposes 38 0.46 28 0.68 10 0.24 Unlocatable 43 0.52 38 0.92 5 0.12 Surveyable 7,838 100.00 3,858 100.00 3,980 100.00 Surveyed 5,278 67.34 2,194 56.87 3,084 77.49 Refusals 1,241 15.83 820 21.25 421 10.58 Absences 1,242 15.85 792 20.53 450 11.31 Inability to respond 77 0.98 52 1.35 25 0.63 Remaining municipalities Selected 15,632 100.00 7,780 100.00 7,852 100.00 Previously selected 27 0.17 19 0.24 8 0.10 Unavailable 31 0.20 21 0.27 10 0.13 Unsurveyable 1,187 7.59 909 11.68 278 3.54 Empty 1,006 6.44 759 9.76 247 3.15 Intended for other purposes 38 0.24 33 0.42 5 0.06 Unlocatable 143 0.91 117 1.50 26 0.33 Surveyable 14,387 100.00 6,831 100.00 7,556 100.00 Surveyed 10,206 70.94 4,182 61.22 6,024 79.72 Refusals 2,273 15.80 1,400 20.49 873 11.55 Absences 1,740 12.09 1,133 16.59 607 8.03 Inability to respond 168 1.17 116 1.70 52 0.69 8
Regarding the failure to update the framework, from the figures in table 1, we can conclude that this is basically due to the empty dwellings, since their number is comparatively much grater than that corresponding to the group of those intended for other purposes and those that are unlocatable, as may also be observed in graph 1. This graph shows the evolution of the percentage of empty dwellings, monthbymonth, in comparison with the percentage of failure to update the framework, due to the dwellings intended for other purposes and the unlocatable dwellings, grouped into the same as other incidences. The percentage of empty dwellings stands at nearly 5.5 percent. Graph 1 Failure to update the framework % (Porcentaje) 8 6 4 2 0 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Months Empty dwellings Other incidences If, in table 1, we compare the data from capitals and the rest of the municipalities, we can observe that the percentage of unsurveyable dwellings is almost three points higher in the rest of the municipalities than in the capitals, with said difference being fundamentally due to the empty dwellings, as can also be observed in graph 2. 9
Graph 2 Empty dwellings % (Percentage) 10 8 6 4 2 Jan Feb Mar Apr May Jun Jul Months Aug Sep Oct Nov Dec Capitals Remaining municipalities Total Graph 3 shows the distribution, throughout the year, of the empty dwellings, distinguishing between the first and second interview. The percentage of empty dwellings is more than twice in the first interview that recorded in the second interview, which is reasonable, as it would seem logical for most of these dwellings to be detected in the first visit made to the selected dwellings. The dwellings that are empty in the second interview correspond to those that, during the year between the first and the second interview, have gone from being inhabited to being empty, or to dwellings that are in a second theoretical collaboration, but in a first real collaboration, and therefore are detected as empty at that moment, since they did not collaborate previously. 10
Graph 3 Empty dwellings % (Percentage) 11 9 7 5 3 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Months 1 First interview Second interview Total IV.5.Nonresponse Nonresponse in a household that resides in a surveyable dwelling may be due to the absence of all of its members, to their refusal to provide collaboration, or to the inability of all of them to fill out the questionnaires or respond to the interviews. Graph 4 shows the evolution of nonresponse, monthbymonth, and in which we can observe that, of the three components of nonresponse, it is the refusals that carry the greatest weight, followed by absences, with the dwellings with an inability to respond being practically null, due to their scarce quantitative importance. It is observed, likewise, that in the third quarter, this period coinciding with the summer holidays, the percentage of absences increases considerably, and particularly during the month of August, where the maximum of absences is reached, exceeding that of refusals. 11
Graph 4 Nonresponse % 25 20 15 10 5 0 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Months Absences Inability to respond Refusals Returning to table 1, It can be observed that, globally, nonresponse represents 30 percent of the surveyable dwellings, with the percentage being almost four points higher in the capitals than in the rest of the municipalities, this difference being mainly due to the absences. In line with this data, we observe that the percentage of surveyable dwellings is nearly four points higher in the remaining municipalities than in the capitals. Graph 5 represents, monthbymonth, the breakdown of the refusals in the provincial capitals and in the remaining municipalities. Although appreciable differences are observed between the percentages of refusals obtained in both type of municipality in certain months, globally, these percentages are equal, as can be confirmed in table 1. 12
Graph 5 Refusals % (Percentage) 22 20 18 16 14 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Months 12 Capitals Remaining municipalities T t l Another breakdown of the refusals is shown in graph 6, in this case between the first and the second interview. It can be observed that the percentage of refusals is more than nine points higher in the first interview than in the second interview, which may be explained by the fact that it seems natural to refuse to collaborate on the first contact with the interview, more than in the second interview, after having collaborated a first time. 13
Graph 6 Refusals % (Percentage) 23 18 13 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Months 8 First interview Second interview Total Regarding the absences, table 1 shows that their percentage is almost four points higher in the capitals that in the rest of the municipalities, the same way that it is nine points higher in the first interview than in the second. The breakdown of the absences in capitals and in the rest of the municipalities, on the one hand, and in the first and the second interview, on the other hand, is represented in graphs 7 and 8, respectively. 14
Graph 7 Absences % (Percentage) 32 26 20 14 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Months 8 Capitals Remaining municipalities T t l Graph 8 Absences % (Percentage) 32 25 18 11 4 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Months First interview Second interview Total 15
IV.6. Incidences in the sample by Autonomous Community Table 2 presents the percentage distribution of the incidences in the theoretical sample, by Autonomous Community. Regarding the failure to update the framework, worth noting are Madrid and País Vasco as the Autonomous Communities with the lowest percentages of unsurveyable dwellings (3.5 percent for the former and 3.8 percent for the latter), that is, with the least failure to update the framework. At the opposite end of the spectrum, CastillaLa Mancha is the Autonomous Community with the greatest failure to update the framework, with 9.7 percent of unsurveyable dwellings. If we now analyse nonresponse, breaking it down into its three components, refusals, absences and inabilities to respond, we will observe that País Vasco, with 27.4 percent, is the Autonomous Community with the highest percentage of refusals. Among the Communities with the fewest refusals, worth noting is Cantabria, with 5.4 percent. With regard to absences, of not is Madrid, with 21 percent, as the Community with the highest percentage thereof. At the other extreme is Cantabria, likewise the Community with the lowest percentage of absences, standing near 10 percent. Regarding the inabilities to respond, outstanding was Navarra, with 2.5 percent, as the Community with the highest percentage, whereas Castilla y León, with 0.4 percent, was the Community with the lowest percentage. Considering lastly the total nonresponse, it may be observed that, on a national level, it represents 30.3 percent of the total surveyable dwellings, that is, five points lower than the value reached in 2007. By Autonomous Community, País Vasco and Madrid are the Communities with the highest percentages, above 40 percent, whlie Cantabria records the lowest percentage, with 16 percent. 16
2. Percent distribution of th incidences in the theoretical sample, by Autonomous Community Autonomous Community (Next) Incidences in the dwellings Total Unsurveyable Previously selected Unavailable Surveyable TOTAL 100.00 6.58 0.14 0.17 93.11 Andalucía 100.00 8.33 0.36 0.12 91.20 Aragón 100.00 7.32 0.10 0.10 92.48 Asturias (Principado de) 100.00 4.40 0.00 0.00 95.60 Balears (Illes) 100.00 7.74 0.11 0.44 91.71 Canarias 100.00 8.87 0.17 0.51 90.44 Cantabria 100.00 7.69 0.00 0.00 92.31 Castilla y León 100.00 7.75 0.33 0.39 91.54 CastillaLa Mancha 100.00 9.66 0.31 0.16 89.87 Cataluña 100.00 4.94 0.00 0.04 95.02 Comunidad Valenciana 100.00 7.19 0.00 0.17 92.64 Extremadura 100.00 8.24 0.19 0.10 91.48 Galicia 100.00 6.42 0.28 0.07 93.23 Madrid (Comunidad de) 100.00 3.53 0.00 0.17 96.30 Murcia (Región de) 100.00 4.33 0.00 0.20 95.47 Navarra (Comunidad Foral de) 100.00 6.52 0.20 0.07 93.22 País Vasco 100.00 3.82 0.05 0.32 95.81 Rioja (La) 100.00 6.79 0.13 0.00 93.09 Ceuta and Melilla 100.00 8.08 0.00 0.00 91.92 (End) Autonomous Community Incidences in the households of the surveyable dwellings Nonresponse Total Surveyed Refusals Absences Inability to respond Total TOTAL 100.00 69.67 15.81 13.42 1.10 30.33 Andalucía 100.00 71.27 16.36 11.60 0.78 28.73 Aragón 100.00 70.28 16.27 12.91 0.54 29.72 Asturias (Principado de) 100.00 72.64 12.69 13.18 1.49 27.36 Balears (Illes) 100.00 73.13 10.23 14.74 1.90 26.87 Canarias 100.00 69.62 14.25 14.91 1.23 30.38 Cantabria 100.00 83.89 5.42 9.86 0.83 16.11 Castilla y León 100.00 76.32 12.73 10.60 0.36 23.68 CastillaLa Mancha 100.00 66.70 16.70 15.91 0.70 33.30 Cataluña 100.00 64.86 19.15 14.54 1.45 35.14 Comunidad Valenciana 100.00 70.48 15.39 12.36 1.76 29.52 Extremadura 100.00 73.09 14.76 11.52 0.63 26.91 Galicia 100.00 76.20 12.50 10.63 0.67 23.80 Madrid (Comunidad de) 100.00 59.66 18.78 20.97 0.59 40.34 Murcia (Región de) 100.00 77.19 8.88 13.00 0.93 22.81 Navarra (Comunidad Foral de) 100.00 70.68 16.38 10.50 2.45 29.32 País Vasco 100.00 58.34 27.37 13.19 1.11 41.66 Rioja (La) 100.00 70.70 12.52 15.54 1.24 29.30 Ceuta and Melilla 100.00 67.78 12.13 18.83 1.26 32.22 17
V. Assessment survey The quality assessment survey of the HBS has a dual objective: To monitor the work of the interviewers who are involved in the HBS To assess the quality of the results To this end, we have followed a mathematical model compiled by the Census Office of the United States, due to Hansen, Hurwitz and Bershad, based on the repeat interview. The operating procedure, very simple, consists of repeating the interviews in a sample of the dwellings selected for the original survey. Subsequently, the data obtained on both occasions is compared, for the purpose of studying the inconsistencies, and quantifying the errors, through the application of different quality indices. The model of Hansen, Hurwitz and Bershad assumes that, in the second interview, or repeat interview, we obtain the true values of the characteristics being studied. Although in practice it is difficult to prove whether or not this objective has been achieved, the data from the repeat interview, obtained with more means and betterprepared interviewers, is assumed to be of a superior quality than the data from the original interview, and will enable basing on it all of the calculations of errors and biases. The comparison of the results obtained from the original interview (O.I.) with those obtained in the repeat interview (R.I.) enables assessing two large types of error that are not sampling errors, which affect the quality of the results: a) Coverage errors, produced by the erroneous omission or inclusion of units in the original survey. b) Content errors, which affect the characteristics studied of the surveyable persons. The fieldwork is carried out by specialised interviewers who conduct the repeat interview at most fifteen days after the original interview, with the data from both interviews referring to the same period of time. V.1. Sample selection As mentioned at the beginning, one of the objectives of the assessment survey is to control the work of the interviewers, and for this purpose, it has been foreseen to inspect, throughout the year, at least one section assigned to each one of them. For the purpose of facilitating the sample selected of R.I., the sections of the survey sample (O.I.) have been organised in blocks, understanding a block to be the quota of annual work that each interviewer has assigned, consisting of thirteen sections, which must be carried out at a pace of one section every four weeks. For reasons regarding cost/section visited, neither Ceuta nor Melilla is studied in R.I.. Solely for the purposes of the selected of the sample of R.I., the sections of 18
the sample from O.I. (except Ceuta and Melilla) have been structured in 181 blocks, with the objective of studying in R.I. one section from each one of them, throughout the year. The survey is distributed, each year, into 26 twoweek periods, 13 of which are called odds (01, 03,..., 23, 25) and the other 13 of which are called evens (02, 04,, 24, 26). Of the 181 blocks to be inspected, 92 of them are performed in the even twoweek periods, and the remaining 89 in the odd twoweek periods. With each one of these two sets of blocks, we have formed 13 itineraries or zones, that is, there are 13 zones for the even twoweek periods, and another 13 zones for the odd twoweek periods, in such a way that in total, there are 26 zones to be investigated in the 26 twoweek periods of the year, and therefore, the selection of the sample is done for the entire year. Each zone is comprised of approximately seven blocks. By random selection, without replacement, one of the zones is made to correspond to each twoweek period, making the selection independently for the even and the odd twoweek periods, for the previously mentioned reason. Each twoweek period, the corresponding section from each block is investigated. In each one of the sections selected for the assessment survey, the interviewers will fill out a quality assessment questionnaire, designed to that effect, for all of the original dwellings that were surveyed in the original interview, as well as for the reserve dwellings that have replaced original dwellings with incidences. For the original dwellings that were not surveyed in the original interview, we will only check the incidence noted in said interview in the work order. According to this selection scheme, the average number of dwellings selected oscillates around 7.5% of the sample of the HCBS. All of the results regarding the assessment survey will be provided on an annual level. V.2. Analysis of the incidences Table 3 includes the distributions of the dwellings and households visited, both in the assessment survey (henceforth, the R.I.) and in the original interview (henceforth, the O.I.), according to the type of incidence. It should be considered that, of the dwellings selected in R.I. to be interviewed, not all can be visited in practice, since, due to different reasons in the organisation of the fieldwork, there are almost always sections, and therefore the dwellings selected therein, that end up not investigated. On comparing the data from both distributions, we can observe that the nonresponse (calculated with regard to the number of surveyable dwellings) is very similar in both cases, as it is barely one percentage point higher in O.I. than in R.I., which breaks with the traditional trend of previous years, in which it was always higher in R.I.. This small difference is due to the percentage represented by those households with an inability to respond, higher in O.I., as the higher percentage of absences 19
in R.I. is compensated by the higher percentage of refusals in O.I.. Regarding absences, it is appropriate to emphasize that the R.I. agents conduct their interviews with greater time limitations, given that as they do not reside in the province, they spend less time in the section, which makes the number of interviews increase. The difference in the percentage of unsurveyable dwellings is small, somewhat less than one piont, though this percentage is also greater in O.I.. 3. Distribution of the dwellings / households visited in R.I. and O.I., according to the type of incidence Dwellings / households visited R.I. O.I.(*) Total Percentage Total Percentage Type of incidence 1,183 100.00 23,836 100.00 Unsurveyable 70 5.92 1,571 6.59 Unavailable 0 0.00 40 0.17 Surveyable 1,113 100.00 22,225 100.00. Surveyed 787 70.71 15,484 69.67. Nonresponse 326 29.29 6,741 30.33 Refusals 106 9.52 3,514 15.81 Absences 219 19.68 2,982 13.42 Inability to respond 1 0.09 245 1.10 (*) En E.O., we have excluded the previously selected dwellings, given that in R.I., this type of incidence is not considered Table 4 includes the coincidences and discrepancies in terms of the coverage of dwellings, between O.I. and R.I., in absolute and percent values. From the analysis thereof, we can arrive at the very high coincidence between the two, which is reflected in the gross difference (indicator of the percentage of error), whose value stands at 0.42 percent. 20
4. Errors of coverage of dwellings Dwellings Total Percentage VISITED IN R.I. 1,183 100.00 Surveyable in O.I. and in R.I. 1,113 94.08 Surveyable in O.I., but not in R.I. (1) Surveyable in R.I., but not in O.I. (2) Unsurveyable in both O.I. and R.I. 5 0 65 0.42 0.00 5.49 Net difference: (1) (2) Gross difference: (1) + (2) 5 5 0.42 0.42 In the dwellings surveyed in R.I., it is generally not possible to use all of the information to assess the content errors, given that some of them have not been surveyed in O.I., due to the different causes included in table 5. As can be observed in this table, in the year 2008, all of the households surveyed in R.I. were likewise surveyed in O.I., and therefore, all of the information collected in R.I. can be used to assess the quality of the survey. 5. Incidences in O.I. of the households surveyed only in R.I. Households Total Percentage Surveyed in R.I. 787 100.00 Surveyed in R.I. and O.I. 787 100.00 Surveyed only in R.I. 0 0.00 Not visited in O.I. 0 0.00 Refusals in O.I. 0 0.00 Absences in O.I. 0 0.00 Not surveyed due to other causes in O.I. 0 0.00 The questionnaires that are processed electronically, and which allow for carrying out the analysis of the errors of coverage of persons and of the content errors in the different characteristics of the survey, are only those corresponding to the dwellings that have been interviewed in both the R.I. and the O.I. Table 6 presents the data regarding the identity of the informant, obtained in the dwellings in which the two interviews were conducted. In O.I., in 50 percent of the dwellings, the data was obtained from the main breadwinner, whereas in R.I., the data was obtained from this person in 53 percent of the cases. The information was provided by the same person in the two interviews in 64 percent of the dwellings. 21
6. Data on the identity of the informant Dwellings Total Percentage Surveyed in O.I. and in R.I. 689 100.00 Informant in O.I. No data recorded Main breadwinner 345 50.07 Another person 344 49.93 Informant in R.I. No data recorded Main breadwinner 367 53.27 Another person 322 46.73 Same informant in O.I. and in R.I. 441 64.01 The fact that the number of dwellings surveyed in O.I. and in R.I. that appears in tables 5 and 6 does not coincide in general, is due to the use of different sources for obtaining it. Table 5 is obtained from the summary of the coverage sheets collected in the fieldwork, whereas the data from table 6 is obtained from the O.I and R.I. questionnaires, once they have been subjected to the electronic processing. VI.Coverage of persons The persons who reside in dwellings in which it has been possible to conduct the interview both for the original survey and for the assessment survey, are always classified in one of the three following categories: Comparable persons Omitted persons Persons erroneously included Comparable persons are those persons whom both agents have considered surveyable. For these persons, we therefore have information from O.I. and from R.I. Omitted persons are those persons whose data has been collected by the R.I. agent, on considering them surveyable, but for whom information does not exist in the O.I. Persons erroneously included are those persons who appear in the questionnaire of the original survey, and whom the R.I. agent has not included in the assessment survey, due to not considering them surveyable. 22
Both the omissions and the erroneous inclusions are considered errors in the coverage of persons, based on the hypothesis that the information of the repeat interview is of a better quality than that of the original interview. The assessment of the coverage of persons is based solely on the occupants of the surveyable dwellings in which both the O.I and the R.I. have been conducted, and the corresponding data may be viewed in table 7. 7. Coverage of persons aged 16 years old and over Persons Total Percentage Interviewed in R.I. 2,009 100.00 Comparable 1,993 99.20 Omitted (1) 16 0.80 Interviewed in O.I. 1,996 99.35 Comparable 1,993 99.20 Erroneously included (2) 3 0.15 Net difference (2) (1) 13 0.65 Gross difference (2) + (1) 19 0.95 The net and gross differences are presented in said table, interpreting the former to be an indicator of the bias, and the latter to be an indicator of the total errors committed. Both differences stand within very low values, and therefore, the coverage of persons can be considered good. Tables C.P.1 to C.P.6 of the annex include the distributions of the persons omitted and erroneously included, by sex, age, marital status and relationship with economic activity. VII.Content errors VII.1. Presentation of results Content errors are analysed from the information supplied, in the two interviews, by the households (or persons) classified as comparable. The O.I and R.I. questionnaires of these households are compared electronically, determining to what extent the two data series differ. To facilitate the analysis, two types of table are compiled: coincidence tables and quality indicator tables. 23
For a characteristic C with K modalities, the coincidence table responds to the following general format: R.I. O.I. Total househol ds M 1 M 2... M j... M k Total households n n. 1 n. 2... n.j... n.k M 1 n 1. n. 11 n. 12... n. 1j... n 1k M 2 n 2. n. 21 n. 22... n. 2j... n 2k.................................................................... M i n i. n i1 n i2... n ij... n ik................................................................... M k n k. n k1 n k2... n kj... n kk n ij represents the number of households/persons classified in modality M i according to the R.I., that in O.I. had been classified in modality M j. Appearing in the main diagonal is the number of households/persons classified identically in both interviews in each modality. These tables allow for studying the transfers of households/population between modalities, due to content errors. From the coincidence table, we can extract, for each modality M i of characteristic C, a dualentry table as shown below: R.I. O.I. With Modality M i Without Modality M i Total With Modality M i a b a + b Without Modality M i c d c + d TOTAL a + c b + d n where: n the total households/persons classified in both interviews, with regard to the reference characteristic. a the number of households/persons classified in modality M i in both interviews. b the number of households/persons classified in modality M i in R.I. and in another in O.I. c the number of households/persons classified in modality M i in O.I and in another in R.I. 24
d the number of households/persons not classified in M i in either of the interviews. Based on this reduced table, the following quality indicators are defined: A. Percentage identically classified P.I. C. (M ) i = a a + b. 100 Varies from zero to one hundred. Indicates response stability. Its optimum value, one hundred, expresses that all households (or persons) belonging, according to R.I., to modality M i, are classified in the same way in O.I. B. Net change index I. C.N. (M) i = c b a + b.100 This may be positive (c>b) or negative (c<b). It measures the response bias of the survey, expressed as a percentage of the number of households belonging to M i, according to R.I. Given that, for its calculation, it does not consider the different weighting of the data in each stratum, this index can only be interpreted as an indicator of the bias, and not as an estimator. C. Rate of net difference T.D.N. (M ) i = c b. 100 n D. Gross change index I. C.B. (M ) i = c + b a + b.100 It may be nonexistent or positive. This indicates the response variance, expressed as a percentage of the number of households belonging to Mi in the R.I. It serves as a measurement of the errors which have been made in this modality. 25
E. Rate of gross difference T.D.B. (M ) i = c + b.100 n From the definition of these indicators, we conclude that, if there are no content errors in a modality, the P.I.C. takes a value of one hundred, and the two indices and the two values take a value of zero. It is also important to note that a small, or even nonexistent, P.I.C. can coexist with a zero bias. This occurs when the errors cancel each other out and b=c. In turn, the I.C.B. can only take on the value of zero if b=c=0, that is, if there is no content error. In order to compare the general quality of the different characteristics assessed, we use the global consistency index, which is defined for a given characteristic C as: I. C. G. (C) = i n n ii. 100 VII.2. Characteristics assessed We have obtained coincidence tables for the following characteristics (section 3 of the annex): A. Of the households Number of persons Dwelling tenancy regime Main source of income Value of net monthly income B. Of the population B.1. All of the population Sex and age B.2. Population aged 16 years old and over Sex and marital status Nationality Highest level of studies completed Relationship with economic activity 26
The quality indicator tables have been obtained for these same characteristics (section 4 of the annex). VII.3. Analysis of the characteristics assessed A. Characteristics of the households A.1. Number of persons The results obtained for this characteristic and the corresponding quality indicators are presented in tables C.1 and I.1 of the annex. The different modalities present high P.I.C.s, as the smallest value, corresponding to the modality of households with 5 persons, stands at 91 percent. The indices of net (indicator of bias) and gross (indicator of the total number of errors) change are quite small, as is customary in this characteristic. A.2. Dwelling tenancy regime Tables C.2 and I.2 of the annex present the results obtained and the quality indicators corresponding to this characteristic. It can be observed that the modality of owned is that which presents the highest P.I.C., with a value of 98.3 percent, and is also the majority modality, since it classifies 85 percent of those classified in R.I.. The lowest P.I.C. is obtained in the modality of granted freeofcharge or semifreeofcharge, with a value of 75 percent, which turns out to be the modality with the lowest number of persons classified. The bias and the gross change index of the owned modality are very small; those of the other modalities are higher, though not especially high. A.3. Main source of income The data related to this characteristic may be seen in tables C.3 and I.3 of the annex. In table C.3, we can observe that most of those classified have been such in one of the first three modalities, that is, selfemployed work, work for others and contributory and noncontributory pensions. The remaining modalities are not representative, given the scarce number of persons classified therein. If we focus on the P.I.C.s (Table I.3), we see that the highest correspond to the majority modalities, work for others and contributory and noncontributory pensions, with values above 87 percent. Regarding the biases and the gross change indices, we have observed that the highest are those that correspond to the minority modalities. A.4. Value of net monthly income The results corresponding to this characteristic are found in tables C.4 and I.4 of the annex. 27
In the C.4 tables, we can see that the quantitatively most important modalities are from the second to the fourth, with 62 percent of the households having been classified in R.I. in the three tables. The fact that income constitutes a characteristic that is collected with great difficulty in surveys, might have an influence on the poor quality of its indicators. Thus, in tables I.4, we can observe that the P.I.C.s are very low, with only the modalities of 500 to 999 euros and 3,000 euros and more being above 50 percent, and the maximum value reached by the latter, 67 percent. In general, the biases are not excessively large, but the gross change indices reach quite significant values. The global consistency indices of the characteristics analysed in this section are included in table 8. It can be observed that the highest indices correspond to the characteristics of number of persons and dwelling tenancy regime, whose values stand above 95 percent. The characteristic that is worst collected, as is customary, is the value of net monthly income with a G.C.I. of 49.6 percent. 8. Global consistency indices Characteristic Number of persons 97.68 Tenancy regime of the dwelling 95.79 Main source of income 84.52 Value of net monthly income 49.64 B. Characteristics of the population B.1. All of the population B.1.1. Sex and age Tables C.5 and I.5 of the annex contain the data corresponding to these characteristics. We observe that the P.I.C.s are very high for both females and males, with the lowest corresponding to the modality of under 16 years of age in females, reaching a value of 88 percent. The highest P.I.C. is that corresponding to the modality of women 20 to 24 years of age, reaching a value of one hundred percent. The biases (N.C.I.) are small, as with the indices of gross change, for both males and females. 28
B.2. Population aged 16 years old and over B.2.1. Sex and marital status The results obtained and the quality indicators corresponding to these characteristics are presented in tables C.6 and I.6 of the annex. Some high P.I.C.s are observed, except in the modality of separated or divorced, which in the case of women stands at 79 percent. This is the modality in which the least persons are classified. Regarding the N.C.I.s and the G.C.I.s, they are very small, with the highest being those that correspond to the mentioned modality. B.2.2. Nationality The results obtained for this characteristic are shown in tables C.7 and I.7 of the annex. We observe that the modality that presents the best indicators is the Spanish modality, in which 95 percent of the persons are classified. Conversely, the modality of both, in which very few persons are classified, is that which presents the worst indicators. B.2.3. Highest level of studies completed The results obtained for this characteristic are shown in tables C.8 and I.8 of the annex. The highest P.I.C. corresponds to the modality of 2nd and 3rd cycle university studies, with 86 percent, whereas the lowest is obtained in VT I, intermediate VT, industrial professional training, standing at 49 percent. Considering these results, it could be said that the P.I.C.s, in general, are low. The net and gross change indices could be considered relatively high, in general, and in particular, those relating to the modality of Elementary Postsecondary education, OSE, Primary education qualification. What is observed in table C.8 is the existence of important population transfers between the different modalities. Thus, of the total classified in R.I. in the modality of Illiterate and without studies, 40 percent are classified in O.I. in the modality of Elementary Postsecondary education, OSE, Primary education qualification, and of those classified in R.I. in this last modality, almost 18 percent have been classified in O.I. in the modality of Illiterate and without studies. These discrepancies may mainly be due to the difficulty in cataloguing a certain level of studies in the classification used thereof, which hampers its encoding. According to the data from table 9, in the year 2008, it is confirmed that there is a break in the trend of persons to elevate their social status by declaring a higher level of studies on receiving the visit of the R.I. interviewer, as what can be observed is that a greater number of persons declares a higher level of studies than in O.I. 29
9. Persons with different levels of studies in the two interviews Level of studies Higher in O.I. 420 Higher in R.I. 172 B.2.4. Relationship with economic activity Tables C.9 and I.9 of the annex contain the data corresponding to this characteristic. Observing table I.9, we see that the modalities that present the worst indicators are those of With work, temporarily absent and Another situation, which are likewise those in which the fewest persons are classified. The remaining modalities have better indicators, though they are not very good, given that the highest P.I.C.s, corresponding to the modalities of Student and Employed, stand at around 90 percent. In turn, table C.9 shows that the customary population transfer between the modalities of Unemployed and Employed is not excessively important on this occasion, since nearly 20 percent of those classified as Unemployed in R.I. are classified as Employed in O.I.. A transfer that is percentually more important than the above is that which occurs between the modalities of With work, temporarily absent and Employed, since 67 percent of those classified in R.I. in the former, are classified in O.I. in the latter. Also important is that which takes place between the modalities of Another situation and Retired (32 percent). What happens is that these transfer have little repercussion as there are very few persons who are classified in the modalities of With work, temporarily absent and Another situation, as previously mentioned. Table 10 presents the global consistency indices regarding the characteristics analysed in sections B.1 and B.2. 10. Global consistency indices Characteristic Age 95.10 Males 96.21 Females 96.89 Marital status 95.71 Males 97.07 Females 96.93 Nationality 98.86 Level of studies completed 63.92 Relationship with economic activity 82.61 30