Working Group Social Protection April Data validation: rules for the core system

Transcription

1 EUROPEAN COMMISSION EUROSTAT Directorate F: Social statistics Unit F-5: Education, health and social protection Luxembourg, 13 April 2016 DOC SP Annex 1-rev 1 ESSPROS Expert Group (1486): Working Group Social Protection April 2016 Data validation: rules for the core system Item 7 of the Agenda Meeting of the Working Group Social Protection Luxembourg, April 2016 BECH Building (Eurostat) Room Quetelet

2 ANNEX 1 OF ITEM 7 DATA VALIDATION RULES FOR THE CORE SYSTEM This Annex lists the validation rules for data collected through the ESSPROS core system data. Note: In this document a cell refers to the actual value of a single observation (described by a set of dimensions). 0. Validation level 0 (format and file structure) Checks that fall within validation level 0 concern the format and the structure of the files transmitted to Eurostat. As the data file structure is currently under discussion (see item 4 of the Working group Agenda) a detailed list will be provided at a later date. DICTIONARIES CHECK A number of dimensions are used to identify individual observations. Certain dimensions, such as those specifying the country and detailed classifications (see Appendix I of the ESSPROS methodology), are fixed according to predefined code lists (or dictionaries). The values used to specify fixed dimensions of the data must conform to the agreed values laid out in the respective code lists (or dictionaries) for each dimension. In the current data structure the coded dimensions are: Country, Item, and Scheme. Please note that period is a dimension identifying an observation but is not coded. When the new data structure (see item 5 of the agenda) is adopted these rules will be updated according to the new dimensions and associated code lists Country codes are valid entries in the country code list (dictionary) The country codes used must be one of the values in the country code list. 0.1 Country codes used are defined in the country code list (dictionary) _ Fatal ERROR 0.1: code for country not defined in the country_list. Examples: = GR Fatal error = FX Fatal error = CR Fatal error 2

3 0.2. Items codes are valid entries in the item code list (dictionary) The items codes used must be one of the values in the item list. 0.2 Item codes used are defined in the item code list (dictionary) _ Fatal ERROR 0.2: code for item not defined in the item_list. Examples: = Fatal error = Fatal error = " " Fatal error 0.3. Scheme codes are valid entries in the schemes code list (dictionary) The scheme codes used must be one of the values in the schemes list. 0.3 Scheme codes used are defined in the scheme code lists (dictionaries) _ Fatal ERROR 0.3: code for scheme not defined in the scheme_list. Examples: = SCH1 Fatal error = allschemes Fatal error = "All Schemes" Fatal error 1. Validation level 1 (checks on the content of the file) 1.1 For each scheme only one criteria is selected for the 5 categories All schemes must be described according to 5 categories of scheme characteristics (Decision making, Legal enforcement, Establishment of entitlements, Scope, Level of protection). For each of these categories only one of the possible criteria must be selected. In the case that a scheme conforms to multiple criteria of a category, the main criteria must be selected (see DOC SP ). 1.1 For each scheme only one criteria is _ Fatal ERROR 1.1: Multiple criteria selected for each of the 5 categories of selected for category scheme characteristics scheme_category 1.2 Cell contents are numbers The cell value must be a number. Any character (".", blank, "*" etc.) reported as value or concatenated to the cell value will generate a fatal error. 1.2 Cell contents are numbers _ Fatal ERROR 1.2: The value in cell is not numeric 3

4 Examples: Cell value = "124 " Fatal error Cell value = "." Fatal error Cell value = "*" Fatal error 1.3 Cell contents are non-negative In general, the cell values should be positive. Any negative value should be explained. 1.3 Cell contents are positive _ Warning ERROR 1.3: The value in cell cell is negative 1.4 Unique cell value check For each cell only one value must be transmitted, no double records are allowed even if they share the same value. 1.4 Unique cell value check _ Fatal ERROR 1.4: cell unique value constraint violated 1.5 For each item the sum of data by scheme is equal to the total For each item, the sum of detailed data by scheme must be equal to the total (All schemes). This rule is valid for all the items except two: Total expenditure ( ): Transfers to other schemes must be subtracted from the sum of data by scheme to obtain the value at all schemes level. Total receipts ( ): Transfers from other schemes must be subtracted from the sum of data by scheme to obtain the value at All schemes level. Separate rules are applied to these cases (see rules 1.6 and 1.7). 1.5 For each item <> and <> the sum of data by scheme = data at all schemes level _ Fatal ERROR 1.5: sum of data by scheme for item doesn t sum up to the total 1.6 For Total expenditure the sum of data by scheme - Transfers to other schemes is equal to the total 1.6 For item the sum of data by scheme _ Fatal ERROR 1.6: sum of data by transfers to other schemes ( ) = data at scheme for Total All schemes level expenditure doesn t sum up to the total 4

5 1.7 For Total receipts the sum of data by scheme - Transfers from other schemes is equal to the total 1.7 For item the sum of data by scheme transfers from other schemes ( ) = data at All schemes level _ Fatal ERROR 1.7: sum of data by scheme for Total receipts doesn t sum up to the total 1.8 Integrity rules linking items are satisfied (Absolute value) The items in the code lists are linked by equations (integrity rules) that must be respected in order to guarantee the consistency of the data (see list below). If an equation is not satisfied, it can be the result of missing components. Such cases should be explained. This check generates a warning if there is any absolute difference greater than The integrity rules defined for the item are Threshold Warning satisfied (aggregate = sum of components) for difference 1 WARNING 1.8: equation eq_number not satisfied for cell Absolute difference greater than 1 Examples (equations expressed with the current codes classification): = = = = Warning 1.9 Integrity rules linking items are satisfied (Ratio) The items in the code lists are linked by equations (integrity rules) that must be respected in order to guarantee the consistency of the data (see list below). If an equation is not satisfied, it can be due the result of missing components. This reason should be explained. This check generates a warning if there is any difference greater than 5%. 1.9 The integrity rules defined for the item are Threshold satisfied (aggregate = sum of components) for ratio Warning 0.05 WARNING 1.9: equation eq_number not satisfied for cell. Difference greater than 5% Examples (equations expressed with the current codes classification): = = = = / = Warning 5

6 Following is the list of integrity rules relating ESSPROS items: Expenditure At scheme level: Total expenditures = Social protection benefits + Administration costs + Other expenditures + Transfers to other schemes At Total schemes level Total expenditures = Social protection benefits + Administration costs + Other expenditures For all schemes (detailed and Total) Transfer to other schemes = Social contributions rerouted to other schemes + Other transfers to other resident schemes Social contributions rerouted to other schemes = rerouted benefits function(i) where i in (sickness, disability, old age, survivors, family/children, unemployment, housing, social exclusion) Other expenditure = Property income + Other For social protection benefits (Total and by function except for Housing), the following identities hold: Social protection benefits = NMT + MT NMT = Cash NMT + Kind NMT Cash NMT = Periodic NMT + Lump sum NMT MT = Cash MT + Kind MT Cash MT = Periodic MT + Lump sum MT Specific additional rules by function: Sickness Cash periodic = Paid sick leave + Other cash period benefits Cash Lump sum = Other cash lump sum benefits Kind = In-patient care + Out-patient care + other benefits in kind In-patient care = Direct provision + Reimbursement Out-patient care = Direct provision of pharmaceutical products + other direct provision + reimbursement of pharmaceutical products + other reimbursement Disability Cash periodic = Disability pension + Early retirement benefit due to reduced capacity of work + Care allowance + Economic integration of the handicapped + Other cash periodic benefits Cash lump sum = Care allowance + Economic integration of the handicapped + Other cash lump benefits Kind = Accommodation + Assistance in carrying out daily tasks + Rehabilitation + Other benefits in kind 6

7 Old age Cash periodic = Old age pension + Anticipated old age pension + Partial pension + Care allowance + Other cash periodic benefits Cash lump sum = Other cash lump sum benefits Kind = Accommodation + Assistance in carrying out daily tasks + Other benefits in kind Survivors Cash periodic = Survivors pension + Other cash periodic benefits Cash lump sum = Death grant + Other cash lump benefits Kind = Funeral expenses + Other benefits in kind Family/children Cash periodic = Income maintenance in the event of child birth + Parental leave benefit + Family or child allowance + Other cash periodic benefits Cash lump sum = Birth grant + Parental leave benefit + Other cash lump benefits Kind = Child day care + Accommodation + Home help + Other benefits in kind Unemployment Cash periodic = Full unemployment benefit + Partial unemployment benefit + Early retirement benefit for labour market reasons + Vocational training allowance + Other cash periodic benefits Cash lump sum = Vocational training allowance + Redundancy compensation + Other cash lump sum benefits Kind = Mobility and resettlement + Vocational training + Placement services and job-search assistance + Other benefits in kind Housing Benefits = MT MT = Kind Kind = Rent benefits + Benefit to owner-occupiers Rent benefits = Social housing + Other rent benefit Social exclusion Cash periodic = Income support + Other cash periodic benefits Cash Lump sum = Other cash lump sum benefits Kind = Accommodation + Rehabilitation of alcohol and drugs abusers + Other benefits in kind 7

8 Receipts: At scheme level: Total receipts = Social contributions + General government contributions + Transfers from other schemes + Other receipts At Total schemes level Total receipts = Social contributions + General government contributions + Other receipts For all schemes (detailed and Total) Social contributions = Employers' social contributions + Social contributions by the protected persons Employers' social contributions = Actual + Imputed Actual = Actual Corporations + Actual Central government + Actual State and local government + Actual Social security funds + Actual Households + Actual Non-profit institutions serving households + Actual Rest of the World Imputed = Imputed Corporations + Imputed Central government + Imputed State and local government + Imputed Social security funds + Imputed Households + Imputed Non-profit institutions serving households + Imputed Rest of the World Social contributions by the protected persons = Employees + Self-employed + Pensioners and others Employees = Employees Households + Employees Rest of the world Self-employed = Self-employed Households + Self-employed Rest of the world Pensioners and others = Pensioners and others Households + Pensioners and others Rest of the world General government contributions = Earmarked taxes + General revenue Earmarked taxes = Earmarked taxes Central government + Earmarked taxes State and local government + Earmarked taxes Social security funds General revenue = General revenue Central government + General revenue State and local government + General revenue Social security funds Transfer from other schemes = Social contributions rerouted from other schemes + Other transfers from other resident schemes Social contributions rerouted from other schemes = Social contributions rerouted from other schemes Households + Social contributions rerouted from other schemes Rest of the World Other receipts = Property income + Other Property income = Property income Corporations + Property income Central government + Property income State and local government + Property income Social security funds + Property income Households + Property income Non-profit institutions serving households + Property income Rest of the World Other = Other Corporations + Other Central government + Other State and local government + Other Social security funds + Other Households + Other Non-profit institutions serving households + Other Rest of the World 1.10 Expenditures and receipts should be balanced Expenditures and receipts may not balance in each calendar year. However, as a long term average, they should be balanced. As consequence the difference in absolute value should not exceed 10% The difference between expenditure and < 10% Warning WARNING 1.10: The receipts in absolute value should not exceeds balance between 10 % expenditure and receipts exceeds 10% (exp-rec)/exp < 10% 8

9 1.11 Social contributions rerouted to/from other schemes should balance to 0 at all schemes level 1.11 Transfer and re-routed social _ Warning WARNING 1.11: The contributions should balance to 0 at all balance between schemes level transfers to other Items : = 0 schemes and from other schemes is not 0 2. Validation level 2 (intra-domain, intra-source) 2.1 Time series checks: outliers detection The time series checks are performed at scheme level. The range of years analysed in the time series check can varies according to the tool used. A 4 years range seems a reasonable choice. The current threshold is roughly fixed to 10%. An analysis of core system data is underway to help define thresholds for each specific item (possibly by country). 2.1 Time series checks: outliers detection 10% Warning WARNING 2.1: Value in cell is a potential outlier 2.2 Revision checks: compare the revised values with the values previously transmitted The revision checks are performed at scheme level. Any cases where a value within a scheme is revised by more than the threshold should be explained. The current threshold is roughly fixed to 10%. This check generates a warning. 2.2 Revision checks: compare the revised values with those previously transmitted 10% Warning WARNING 2.2: Value in cell is significantly different from data provided previously. 9