Hands-on exercise Data Verification

Transcription

1 Hands-on exercise Data Verification Review the following documents: 1. Data Verification Tool for the health facility level (Annex 1) 2. Data Verification Tool for the District level (Annex 2) 3. Analyses recommended in the DQR Toolkit (Annex 3). Targeted DQR Reporting Template (Annex ) Discuss among your group and answer the following questions: 1. How is the tool/process similar to that used in the OSDV? 2. How is the tool/process different from that used in the OSDV? 3. What does the LFA see as the 3 main challenges in conducting the Data Verification as part of the Targeted Data Quality Review?. How would the LFA overcome these 3 main challenges? - Time allocated for the exercise: 30 minutes - Time allocated for feedback to the group: 5 minutes per group Document Title, 00 Month 20XX 1

2 Annex 1: Health Facility Data Verification Malaria Number Question Result Skip MALARIA MALARIA CASES DV087 DV088 A DV088 B Does this facility provide malaria diagnosis? YES... 1 NO... 2 Does this facility report malaria cases to a reporting system? YES... 1 NO... 2 What indicator definition is used when reporting malaria cases? TOTAL MALARIA CASES TREATED (PRESUMED AND CONFIRMED)... 1 CONFIRMED MALARIA CASES (BY EITHER MICROSCOPY OR RDT)... 2 SUSPECTED MALARIA CASES TESTED 3 OTHER (SPECIFY) 96 NO YES DV089 Which of the following reporting system(s) does the facility report malaria cases to: 01 Health Management Information System National malaria programme Non-governmental organizations or 1 2 institutions 0 Other reporting system 1 SPECIFY 2 SOURCE DOCUMENTS AND REPORTS DV090 Which of the following documents are NO A used at this facility to record the number of malaria cases? YES 01 OPD register Tally sheets Patient cards Lab register 1 2 DV09 5 DV Other 1 2 Document Title, 00 Month 20XX 2

3 DV090 B What is the source document used by this facility for monthly reporting of malaria cases? We are primarily interested in the main document that is used for compiling monthly summary statistics for malaria cases. Please report if any improvised documents are used. SPECIFY OPD REGISTER... 1 TALLY SHEETS... 2 PATIENT CARDS... 3 OTHER (SPECIFY) BASED ON RESPONSE TO QUESTION DV090B, PLEASE ASK THE PERSON IN THE FACILITY WHO REGULARLY PREPARES THE FACILITY MONTHLY REPORTS TO PROVIDE YOU WITH THE SOURCE DOCUMENT USED TO COMPILE AND SUMMARIZE INFORMATION FOR MONTHLY REPORTING (i.e. REGISTERS, TALLY SHEETS, ETC.) AS WELL AS THE MONTHLY REPORTS FOR MONTH1, MONTH2, AND MONTH3 FOR MALARIA CASES RECORDED IN THE FACILITY. REVIEW SOURCE DOCUMENT FOR MALARIA CASE RATE AND ANSWER THE FOLLOWING QUESTIONS DV091 Please confirm the availability of the source document for malaria cases for Month1 to Month3. If available, please recount the number of malaria cases as recorded in the main source document for Month1 to Month3. E AND COMPLET E (A) Source document available E BUT PARTLY* COMPLET E E BUT NO DATA RECORDE D NO (B) Recount number of malaria cases in source document 01 Month1 1 B 2 B Month2 1 B 2 B Month3 1 B 2 B 3 DV092 DV092 *PARTLY: the source document is available but some information is missing REVIEW MONTHLY REPORT FOR MALARIA CASE RATE AND ANSWER THE FOLLOWING QUESTIONS DV092 Please confirm the availability of the monthly report for malaria cases for Month1 to Month3. If available, please record the number of malaria cases recorded in the monthly report for Month1 to Month3. (A) Monthly report available (B) Record number of malaria cases in monthly report E AND COMPLET E E BUT PARTLY* E BUT NO DATA NO Document Title, 00 Month 20XX 3

4 COMPLET E RECORDE D 01 Month1 1 B 2 B Month2 1 B 2 B Month3 1 B 2 B 3 DV093 DV093 *PARTLY: the source document is available but some information is missing DISCREPANCIES DV093 What are the reasons for the discrepancy (if any) observed between the main source document and the monthly report? NO DISCREPANCY... 1 DATA ENTRY ERRORS... 2 ARITHEMATIC ERRORS... 3 INFORMATION FROM ALL SOURCE DOCUMENTS NOT COMPILED CORRECTLY..... SOURCE DOCUMENT AND/OR MONTHLY REPORT NOT E.. 5 OTHER (SPECIFY) Annex 2: Data Verification District Level Number Question Result Skip DATA VERIFICATION DISTRICT LEVEL REPORTING PERFORMANCE PLEASE COMPLETE FOR THE MOST RECENT MONTH OF THE SELECTED REPORTING PERIOD DVD00 6 How does the district handle data from health facilities? Aggregates indicators into district totals and sends a paper report to the next level... 1 Enters data by facility into a database and the data are sent electronically to the next level... 2 Enters district totals into a database and the data are sent electronically to the next level... 3 Health facilities send data directly to the national level and the district has access to the data... The district does not have access to the data... 5 THE FOLLOWING SECTION IS EXAMINING REPORTING COMPLETENESS OF THE MONTHLY (OR QUARTERLY) REPORTS THAT ARE RECEIVED FROM HEALTH FACILITIES THAT INCLUDE INFORMATION ON THE SELECTED DATA VERIFICATION INDICATORS. IF THE INFORMATION ON THE SELECTED INDICATORS ARE IN DIFFERENT FORMS, THE COMPLETENESS OF DIFFERENT FORMS IS EXAMINED. IF THE SELECTED INDICATORS ARE ALL REPORTED ON THE SAME MONTHLY REPORT, QUESTIONS DVD007-DVD009 WILL BE THE SAME FOR ALL INDICATORS. HOWEVER, QUESTION DVD010 LOOKS AT THE COMPLETENESS OF THE INDICATOR ITSELF. SO WHILE THE FORM MIGHT BE COMPLETE AND SUBMITTED ON TIME, INFORMATION ON THE ACTUAL INDICATOR MIGHT BE MISSING. FOR EXAMPLE, IF IMMUNIZATION AND ANTENATAL CARE ARE ON THE SAME FORM, THEIR MONTHLY REPORTING COMPLETENESS WILL BE THE SAME. HOWEVER, IF THE INFORMATION ON DOSES OF VACCINES GIVEN (E.G. DTP/PENTA) IS MISSING, THE QUESTION ON INDICATOR COMPLETENESS (DVD10) WILL BE DIFFERENT. Document Title, 00 Month 20XX

5 Please complete the following for each of the selected indicator. IF SOME OF THE INDICATORS ARE REPORTED ON THE SAME FORM, QUESTIONS (DVD007-DVD009) WILL HAVE THE SAME ANSWERS FOR THEM. Program area 1 Program area 2 Program area 3 Program area Program area 5 DVD00 7 DVD00 8 How many reports should there have been from all health facilities for month1 to month 3? (Adapt in country) DVD007a Value DVD007 b Value How many reports are there? DVD008a DVD008 b DVD007c Value DVD008c DVD007 d Value DVD008 d DVD007e Value DVD008e DVD00 9 DVD01 0 Check the dates on the reports received. How many reports were received on time? (i.e., on time means that the report was received by the due date) How many reports were complete? (i.e., complete means that the report contains the reported count relevant to the indicator Value DVD009a Value DVD010a Value Value DVD009 b Value DVD010 b Value Value DVD009c Value DVD010c Value Value DVD009 d Value DVD010 d Value Value DVD009e Value DVD010e Value Number Question Result Skip DATA VERIFICATION: PLEASE VERIFY UP TO FIVE INDICATORS, ONE FROM EACH OF FIVE PROGRAM AREAS, FROM THE MOST RECENT MONTH OF THE SELECTED REPORTING PERIOD [For EACH INDICATOR BEING ASSESSED]: INDICATOR 1: DVD01 1 DVD01 2 Re-aggregate the numbers from the reports received from all health facilities in the district. What is the re-aggregated number? What aggregated result was contained in the summary report prepared by the district (and submitted to the next reporting level)? Value Value Document Title, 00 Month 20XX 5

6 DVD01 3 What are the observed reasons for the discrepancy (if any)? (i.e., any data entry errors, arithmetic errors, missing source documents, other reason). No discrepancy... 1 Data entry errors... 2 Arithmetic errors... 3 Information from submitted reports not compiled correctly..... Monthly reports not available... 5 Other (specify) 96 Re-aggregation of health facility monthly report values INDICATOR VALUES FOR EACH OF THE SELECTED PRIORITY INDICATORS (UP TO 5) BY HEALTH FACILITY FOR THE SELECTED REPORTING PERIOD DVD026: Please enter the values of the indicators for all facilities in the district for the most recent month of the selected reporting period No. (a) Health facility name (b) Facility ID (if available) (c) Indicator continued for all facilities in the district Annex 3: ANALYSIS Excepts from the DQR Toolkit 1. Availability of services (d) Indicator 2 (e) Indicator 3 (f) Indicator (g) Indicator 5 The percentage of facilities in the sample providing the specific health services should be included in the presentation of results. This will provide information on the number of facilities on which the subsequent data verification results are based. Table 3.1 shows one example of how the data may be presented. Table 3.1 Percentage of facilities in the sample providing each health service, by stratum Facility type Stratum 2 Overall Hospitals Health centres ANC % % % % % % EPI % % % % % % Malaria % % % % % % TB % % % % % % HIV % % % % % % Note: ANC = antenatal care; EPI = Expanded Programme on Immunization; TB = tuberculosis; HIV = human immunodeficiency virus. Document Title, 00 Month 20XX 6

7 2. Availability of documents for review If a facility offers a particular service, it must also have the source documents (registers, tally sheets, etc.) of the three-month verification period available for review on the day of the data verification survey. The selected programme indicators (and their related services) should have standard Ministry of Health registers, tally sheets or other documents which health facilities are supposed to use to record daily activities. While it is possible that health facilities may use multiple documents to record the services provided, it is important to identify if there is a main source document that is used for monthly reporting. The analysis should include information on which documents are used as the main source for reporting service outputs and which of these documents are available. Table 3.2 shows the distribution of the main source document used by a facility for its monthly reports. Table 3.2 Distribution of main document used by a facility to report service data for monthly reports, by programme area ANC EPI Malaria TB HIV Register N (%) N (%) N (%) N (%) N (%) Patient cards N (%) N (%) N (%) N (%) N (%) Tally sheet N (%) N (%) N (%) N (%) N (%) Improvised document N (%) N (%) N (%) N (%) N (%) Others N (%) N (%) N (%) N (%) N (%) Total N (100%) N (100%) N (100%) N (100%) N (100%) Note: ANC = antenatal care; EPI = Expanded Programme on Immunization; TB = tuberculosis; HIV = human immunodeficiency virus. Table 3.3 shows the percentage availability of these documents for all the three months. % availability of source documents for each facility 1 = n i=1 Available month 1 i + Available month 2 i + Available month 3 i 3n where n is the total number of facilities using that particular source document X 100 Table 3.3 Availability of main document used by a facility to report service data for monthly reports, by programme area ANC EPI Malaria TB HIV Register % % % % % Patient cards % % % % % Tally sheet % % % % % 1 Except in the case of TB which has quarterly reporting rather than monthly reporting. For TB the available source documents will be for one quarter-year instead of for three months. Document Title, 00 Month 20XX 7

8 Improvised % % % % % document Others % % % % % Note: ANC = antenatal care; EPI = Expanded Programme on Immunization; TB = tuberculosis; HIV = human immunodeficiency virus. 3. Match between source documents and monthly reports The number of events recounted from the register should exactly match the number reported in the monthly reporting form. Table 3. shows the percentage match between the reported documents and recounted documents for all the three months. % match between reported and recounted service outputs = n i=1 (Report month 1 i = Recount month 1 i ) + (Report month 2 i = Recount month 2 i ) + (Report month 3 i = Recount month 3 i ) 3n where n is the total number of facilities using that particular source document X 100 Table 3. Percentage of facilities that have an exact match between recounted numbers of main source documents and monthly reports, by facility type Stratum 1 Overall Hospitals Health centres ANC2 % % % Deliveries % % % Measles immunization % % % Outpatient visits % % % Malaria cases (outpatient) % % % Average % % % Note: ANC = antenatal care.. Verification factor Even if the reported and recounted numbers do not match exactly, it is useful to take into account the degree of disparity between the two. For a given indicator, the verification ratio at a facility is computed as the recounted number of events from source documents divided by the reported number of events from the HMIS. Verification factor = Recounted number of events from source documents Reported number of events from the HMIS A verification ratio higher than 1 implies that there is underreporting of events in the HMIS for the verification period. If the verification ratio is less than 1, this implies that there is overreporting of events in the HMIS for the period chosen for the analyses (alternatively, this may be due to missing documents as the services may have been rendered but cannot be verified). For each health facility in the sample, a verification ratio is computed for each of the core data verification indicators. If a particular service is not offered at the facility, the verification ratio cannot be calculated for the corresponding indicator. To obtain a national verification factor for an indicator, a weighted mean is calculated from individual health facility verification ratios. Document Title, 00 Month 20XX 8

9 Annex except from Targeted DQR Reporting Template Data Verification Please provide the results of the data verification for each indicator verified. Results can be copied in from the Targeted DQR Data Verification Excel tool. a. Data verification at facility or site level (use table from excel workbook) A) Assessment of Data Accuracy at Service Delivery Points - Service Delivery Points Data Verification District No. Name of Service Delivery Point Verified Result (V) Reported Result (R) Reason for Variance Verification Factor Weighted Error Absolute Difference # Facilities assessed Total b. Data verification at district level (copy results in from the Targeted DQRData Verification Excel tool) District No. Verified Result (V) Reported Result (R) Reason for Variance Verification Factor (V/R) Weighted Error Absolute Difference # Districts assessed Inference: Describe the sampling used, and the inference possible from the results above (margin of error, confidence interval and/or Pass/Fail) 9

10 10