Managing for Results Fiscal Year 2000 Performance Measures

Transcription

1 Performance Audit Report Managing for Results Fiscal Year 2000 Performance Measures Department of Human Resources Family Investment Administration August 2001

2 This report and any related follow-up correspondence are available to the public and may be obtained by contacting the Office of Legislative Audits or the Department of Legislative Services Office of the Executive Director, 90 State Circle, Annapolis, Maryland. Alternate formats may be requested by contacting the Office of Legislative Audits at , (301) (voice), or (toll-free voice), or (fax), and (Maryland Relay Service). Audit reports can also be viewed or downloaded from the Internet. Our web site address is

3 August 16, 2001 Delegate Samuel I. Rosenberg, Co-Chair, Joint Audit Committee Senator Nathaniel J. McFadden, Co-Chair, Joint Audit Committee Members of Joint Audit Committee Annapolis, Maryland Ladies and Gentlemen: We conducted a performance audit to determine the accuracy of certain fiscal year 2000 Managing for Results performance measures reported by the Department of Human Resources Family Investment Administration in the fiscal year 2002 budget request. We also determined whether adequate control systems were in place over the collection and reporting of related performance measurement data. In accordance with a plan approved by the General Assembly budget committees and the Joint Audit Committee, the performance measures audited were selected by the Department of Legislative Services Office of Policy Analysis. We have developed a system to categorize the results of our audit of performance measures. Measures are designated as either Certified, Certified with Qualification, Factors Prevented Certification or Inaccurate, and are further described in Exhibit 2. Of the four Administration performance measures we audited, two measures were Certified with Qualification and two measures were categorized as Factors Prevented Certification. With respect to the two performance measures categorized as Certified with Qualification, the Administration had not generated and retained detail results of computer matches conducted to determine performance. However, we were able to verify that reported performance was reasonably accurate through alternative means. For the two measures categorized as Factors Prevented Certification, performance reported by the Administration for one measure represented estimated rather than actual performance and for the other measure, the Administration had not established procedures to ensure that all relevant data was included in the performance calculation. We wish to acknowledge the cooperation extended to us during our audit by the Family Investment Administration. Respectfully submitted, Bruce A. Myers, CPA Legislative Auditor

4 (This Page Intentionally Left Blank) 2

5 Background Information In July 1997, the Governor s initiative Managing for Results was implemented to increase governmental accountability by utilizing resources more effectively and efficiently to meet the needs of the State s customers and stakeholders (such as the citizens of Maryland). Managing for Results is a future-oriented strategic planning process that establishes direction and priorities for State programs to achieve meaningful results. The success of a program in achieving desired results is determined through the development and monitoring of program performance measures. Managing for Results was implemented over a three-year period beginning with the fiscal year 1999 budget submission and is linked to the budget and decision making process. Specifically, State agencies are required to submit missions, goals, objectives and performance measures for each program to the Department of Budget and Management as part of their annual budget requests. This information may then be considered in determining Statewide spending priorities and the allocation of resources in agency budgets. The General Assembly budget committees and the Joint Audit Committee approved the plan developed by the Department of Legislative Services whereby the Department s Office of Policy Analysis identifies performance measures in the budget requests of selected agencies to be audited by the Office of Legislative Audits. In accordance with this plan, the Office of Policy Analysis identified fiscal year 2000 performance measures at eight agencies for audit. This report contains the results of our audit of the fifth agency and comprises four measures. We have developed a system to categorize the results of our audit of performance measures. Measures are designated as either Certified, Certified with Qualification, Factors Prevented Certification or Inaccurate. The certification categories are defined in Exhibit 2. These categories were assigned based on a combination of the adequacy of controls over a performance measure, consistency between the performance measure calculation method and the performance measure definition, and testing of source documents. 3

6 Findings and Recommendations Certification Results for Selected Fiscal Year 2000 Performance Measures Performance Measure (See Exhibit 1) Results Reported 1 Level of Certification (See Exhibit 2) Comments Percentage of families that remain independent from Temporary Cash Assistance 60% Factors Prevented Certification The reported performance for fiscal year 2000 was estimated rather than actual performance. (See Finding #1) Percentage of increased earnings over time for employed individuals 50% Certified with Qualification Job retention rate 74% Certified with Qualification The Administration did not generate and retain detail results of computer matches to support reported performance. In addition, the computer match procedure was not as comprehensive as it could be to ensure the accuracy of the match results. However, we were able to verify that reported performance was reasonably accurate through alternative means. (See Finding #2) Job placement rate 40% Factors Prevented Certification Procedures had not been established to ensure that all job placements were included in the performance calculation. In addition, the methodology used by the Administration to calculate performance was not consistent with the measure definition and the description of the measure in the fiscal year 2002 budget submission did not indicate the specific population to be included in the calculation. (See Finding #3) 1 Source: Fiscal Year 2002 Budget Submission 4

7 Finding #1 The Administration did not disclose in the fiscal year 2002 budget submission that the performance reported for one measure for fiscal year 2000 represented estimated rather than actual performance. Analysis The fiscal year 2000 performance reported by the Administration in the fiscal year 2002 budget submission for the measure percentage of families that remain independent from Temporary Cash Assistance represented estimated rather than actual performance. The measure is calculated by dividing the number of cases closed during a fiscal year that remained closed for at least 12 months by the total number of cases closed during the fiscal year. The Administration reports this as actual performance for the fiscal year in which the case closed. The reported fiscal year 2000 performance represents an estimate because the percentage of fiscal year 2000 closed cases that were still closed at least one year later cannot be determined until the end of fiscal year Under this method, the Administration has elected to revise the estimated data previously reported. For example, the fiscal year 2002 budget submission contained actual fiscal year 1999 data that had previously been reported as an estimate. Specifically, the Administration reported that 63 percent of the Temporary Cash Assistance cases closed in fiscal year 1999 remained closed at least 12 months later. Our review of the methodology used to determine the 1999 data concluded that the methodology was not consistent with the measure definition. Specifically, it was not unusual for the same case to close and reopen more than one time during fiscal year 1999, and when this occurred, the Administration counted each time the case closed in the denominator, but if the case remained closed the final time for at least 12 months, it was counted only once in the numerator. Our review of the fiscal year 1999 case closures used by the Administration in calculating performance disclosed that about 10 percent of these closures were cases that closed more than one time during the fiscal year. Our recalculation of the measure for fiscal year 1999, excluding these duplicate cases from the denominator, resulted in the percentage of families that remain independent from Temporary Cash Assistance increasing to 71 percent from the reported performance of 63 percent. Recommendation #1 We recommend that the Administration disclose in future budget submissions performance that is being reported as estimated rather than actual. In addition, we recommend that the methodology used to calculate reported performance be consistent with the measure definition. 5

8 Finding #2 The Administration did not generate and retain detail results of computer matches to support the reported performance for two measures. In addition, the computer match procedure did not consider recipient names. Analysis The Administration did not generate and retain detail results of computer matches to support the reported performance for the measures percentage of increased earnings over time for employed individuals and job retention rate. The performance reported by the Administration for these measures was based on computer matches conducted of Temporary Cash Assistance recipient data and employer reported wage earning data maintained by the Department of Labor, Licensing and Regulation. Summary reports of computer match results were generated and we were able to verify reported performance for the two measures to the summary reports. However, because the Administration did not generate and retain the detail results of the computer matches, we have limited assurance that the summary reports were accurate. At our request, the Administration re-conducted the computer matches and provided us with detail results of the matches. Our comparison of the detail results to the summary reports disclosed only minor discrepancies. Administration personnel advised us that these discrepancies were primarily due to changes in the wage earning data that occurred after the computer matches were initially conducted. Additionally, the computer matches were conducted using only the social security numbers of the Temporary Cash Assistance recipients and the wage earners. The match did not consider the individuals names even though this information was included on both files used to conduct the computer matches. Consequently, there was reduced assurance that the match results were accurate. Our review of 65 matches used in the calculation of the performance measures disclosed that two of the matches had the same social security number but different names on the Temporary Cash Assistance recipient and wage earner files. In this instance, we concluded that the reported performance was still reasonably accurate. Recommendation #2 We recommend that the Administration generate and retain detail results of computer matches to support reported performance. Additionally, we recommend that the Administration enhance the computer match process to include recipient names. 6

9 Finding #3 The Administration had not established procedures to ensure that all job placements were included in the calculation of the job placement rate performance measure. In addition, enhancements were needed in the Administration s calculation methodology and in the description of the measure as reflected in the budget submission. Analysis The Administration defined the performance measure job placement rate as the number of non-exempt Temporary Cash Assistance recipients and applicants placed into unsubsidized employment through the Department s efforts, divided by the average number of Temporary Cash Assistance recipients and applicants. However, the Administration had not established procedures to ensure that all job placements were included in the calculation of the performance measure. Specifically, the Administration identified job placements by reviewing information recorded on its Work Opportunity Management Information System (WOMIS). However, Administration personnel advised us that its ability to identify job placements was limited since recipients do not always advise the Administration that they have obtained employment. Consequently, the actual number of job placements may be significantly higher than that reflected in WOMIS. These additional job placements would result in an increase in the reported performance. Our test of 25 recipients who did not have a job placement recorded in WOMIS during fiscal year 2000 disclosed that 6 recipients had wage earnings recorded for the fiscal year in the Department of Labor Licensing and Regulation s records, thus indicating that a job placement may have occurred without the Administration s knowledge. In addition, the methodology used to calculate the job placement rate was not consistent with the measure, as defined by the Administration. Specifically, the same recipient sometimes was placed in more than one job during the fiscal year and when this occurred, the Administration counted each job placement in the numerator but counted the recipient only once in the denominator. Excluding these multiple placements from the numerator would result in a reduction of the reported performance. Finally, the description of the measure in the fiscal year 2002 budget submission did not clearly indicate that individuals who had applied for but did not receive Temporary Cash Assistance, and who were placed into unsubsidized employment through the Department s efforts, were included in the calculation of the measure. Recommendation #3 We recommend that the Administration establish procedures to help identify all job placements for inclusion in the calculation of the job placement rate performance measure. In addition, we recommend that in future budget submissions the Administration ensure that the methodology used to calculate reported performance is consistent with the measure definition and that the measure description clearly indicates the items included in the calculation of the measure. 7

10 Scope, Objectives and Methodology Scope We conducted a performance audit of selected fiscal year 2000 performance measure results reported by the Department of Human Resources Family Investment Administration to determine whether they were accurately reported and whether adequate control systems were in place over the collection and reporting of data used to measure performance. This audit was conducted under the authority of the State Government Article, Section of the Annotated Code of Maryland and was performed in accordance with generally accepted government auditing standards. The General Assembly budget committees and the Joint Audit Committee approved the plan developed by the Department of Legislative Services whereby the Department s Office of Policy Analysis identifies performance measures in the budget requests of selected agencies to be audited by the Office of Legislative Audits. In accordance with this plan, the Office of Policy Analysis identified fiscal year 2000 performance measures at eight agencies for audit. This report covers our audit of the Family Investment Administration and comprises four performance measures. We have previously reported the results of our audits of four agencies; the audit results of the remaining three agencies will be included in forthcoming reports. Objectives The objectives of our audit were: (1) to determine whether the fiscal year 2000 measurement results for the selected performance measures have been accurately reported; and (2) to determine whether adequate control systems are in place over the collection and reporting of the data related to the measurement results. Our performance audit did not include an assessment of whether the performance measures reviewed are consistent with the goals and objectives of the related programs or are meaningful indicators of program performance. Methodology To accomplish our objectives, we interviewed Administration personnel and reviewed performance calculations for accuracy and ensured that these calculations were consistent with the definition of the performance measure as noted in Exhibit 1. We utilized sampling techniques or other methods as deemed appropriate to test the related source documents to verify the accuracy of the reported measure. We also analyzed the Administration s performance measurement data collection and reporting activities to evaluate whether proper controls were in place. Additionally, we developed a system to categorize the results of our audit of performance measures. The four categories represent varying levels of certification of the accuracy of the fiscal year 2000 performance reported by the Administration. The categories of performance certification are defined in Exhibit 2. 8

11 Our fieldwork was completed during the period from February to May The Administration s responses to our findings and recommendations appears as an appendix to our report. As prescribed in the State Government Article, Section of the Annotated Code of Maryland, we will advise the Administration regarding the results of our review of its responses. 9

12 Exhibit 1 Definition of Performance Measures Audited Performance Measure Definition Percentage of families that remain independent from Temporary Cash Assistance Percentage of increased earnings over time for employed individuals Job retention rate Job placement rate The measure reflects the percentage of those Temporary Cash Assistance cases closed during the current fiscal year that remained closed for at least 12 months. The measure reflects the percentage increase in earnings of Temporary Cash Assistance recipients employed in both a base quarter (such as January to March) and the second subsequent quarter (such as July to September). The percentage increase in earnings for each quarter is then averaged to calculate the percentage increase for the year. This measure reflects the percentage of Temporary Cash Assistance recipients employed in a base quarter who are also employed in the subsequent quarter. The percentages for each quarter are then averaged to calculate the job retention rate for the year. The measure reflects the number of nonexempt Temporary Cash Assistance recipients and applicants (that is, individuals who should be participating in a work activity) who, through the Department s efforts, were placed into unsubsidized employment divided by the average number of non-exempt recipients and applicants. (An unsubsidized employment situation exists when the employer is not receiving a Federal or State subsidy to employ the individual.) 10

13 Exhibit 2 Categories of Performance Certification Category Certified Certified with Qualification Factors Prevented Certification Inaccurate Definition Reported performance was reasonably accurate. Reported performance was reasonably accurate but either minor deficiencies were noted with the supporting documentation, or controls were not sufficient, or the methodology used to calculate reported performance was not consistent with the measure definition. Actual performance could not be verified as documentation was unavailable and/or controls were not adequate to ensure the accuracy of reported results. Reported performance differed significantly from actual performance. 11

14

15 Finding #1: The Administration did not disclose in the fiscal year 2002 budget submission that the performance reported for one measure for fiscal year 2000 represented estimated rather than actual performance. Response: We agree with the recommendation that future budget submission should disclose performance that is being reported as estimated rather than actual. The final budget submission that DHR sent to the Department of Budget and Management (DBM) for SFY 2002 disclosed that the fiscal year 2000 data was an estimate. We have been advised that DBM subsequently established a policy to remove any reference to estimates for the 2000 data. As such, the information reported in the budget book did not reflect DHR s original submission. We have discussed this with our DBM budget analyst to resolve the issue in future years. The methodology used to calculate this measure reflects the realities in the Temporary Cash Assistance caseload. Researchers and program administrators have identified three categories of welfare families: short-term, long-term and churners. Churners are those who periodically or frequently leave the TCA rolls but come back fairly rapidly. Under MFR, FIA identified Goal 1 as to move families with an employable parent and no children under one toward a speedy and lasting exit from TCA. Objective 1.1 is to ensure at least 55% of families leaving TCA remain independent As such, we are measuring our success in helping all cases (short-term, long-term, and churners ) become independent of cash assistance. This means that cases that exit the rolls, regardless of how frequently, are counted each time they leave since our objective is that each exit be lasting. Those that aren t must be counted in the denominator of the performance measure but not the numerator because that instance of case closure did not result in a successful outcome for this measure. For added clarity, the Administration will change the language describing this measure to read: Percentage of TCA case closures that remain closed for 12 consecutive months (# of cases with an employable adult and no children under one that close and do not return to TCA within 12 months compared to the total number of TCA cases that closed). Finding #2: The Administration did not generate and retain detail results of computer matches to support the reported performance for two measures. In addition, the computer match procedure did not consider recipient names. Response: We agree with this finding. Because the wage record archive database is continually updated, we will establish a file with detail records each time we conduct a match of TCA administrative records and wage data records so that point-in-time data is not lost.

16 The Department of Labor, Licensing and Regulation, through its contractor, the Jacob France Center at the University of Baltimore, updates the wage record database to allow for late reported data from employers to be included. Beginning with the match conducted in November 2001 for the fourth quarter of 2000, we will maintain separate history files with detail data from each match. We also agree that adding a secondary check for the individual s name will result in a limited improvement in the reliability of the matched records. The Administration will add names to the computer matching procedures beginning with the match conducted in November 2001 for the fourth quarter of Finding #3: The Administration had not established procedures to ensure that all job placements were included in the calculation of the job placement rate performance measure. In addition, enhancements were needed in the Administration s calculation methodology and in the description of the measure as reflected in the budget submission. Response: The placement measure used for the MFR budget submission relies on data collected by local department staff in the WO MIS system to record individuals who obtained employment through the services provided by the local department. We are aware that the number of TCA customers who become employed may be greater than the number of individuals who are recorded as being placed in a job by the local department. Customers who find jobs through their own initiative may not be considered placed in those jobs by the local department. As such, the data collected in the Maryland Automated Benefit System (MABS) and the State New Hires Directory will not match the data collected in WO MIS. Another issue with the placement measure involves the integrity of the data in WO MIS. When a customer becomes employed, it is likely that he or she will no longer be eligible for Temporary Cash Assistance. As such, it is sometimes difficult to collect the necessary placement documentation since the customer is no longer subject to a penalty for failure to comply with our requirements. However, to maintain the integrity of the data in WO MIS we require that the local department collect sufficient documentation to verify the placement. This results in a Catch 22 where the more selective we are with entering placements in WO MIS based on adequate documentation, the fewer job placements get recorded. Conversely, the more liberal we are with the verification requirements the more placements we can record but the less reliable the data in WO MIS. To resolve some of this problem, we have enhanced our interface between CARES and WO MIS effective April In our daily download routine from CARES to WO MIS, we now include employment data so that we are able to capture any new employment information that is recorded in CARES. We then produce a report for local departments to follow up on the information to determine if a placement should be entered in WO MIS. Local departments also have access to

17 MABS, which contains quarterly wage and Unemployment Insurance data, and the State New Hires Directory, which contains information regarding recently filled jobs. These latter two data sources are based on information reported by employers themselves and are used to identify job placements that may need to be recorded in WO MIS. To address the auditors concerns regarding the calculation methodology and description of the measure, the Administration will change the outcome in the budget submission to read: Job Placement Rate (# of TCA recipients and applicants placed in unsubsidized employment divided by the total number of TCA applicants plus the number of employable adults with no children under one who have received TCA).