Managing for Results Fiscal Year 2000 Performance Measures

Transcription

1 Performance Audit Report Managing for Results Fiscal Year 2000 Performance Measures University System of Maryland University of Maryland, College Park August 2001

2 This report and any related follow-up correspondence are available to the public and may be obtained by contacting the Office of Legislative Audits or the Department of Legislative Services Office of the Executive Director, 90 State Circle, Annapolis, Maryland. Alternate formats may be requested by contacting the Office of Legislative Audits at , (voice), or (toll-free voice), or (fax), and (Maryland Relay Service). Audit reports can also be viewed or downloaded from the Internet. Our web site address is

3 August 27, 2001 Delegate Samuel I. Rosenberg, Co-Chair, Joint Audit Committee Senator Nathaniel J. McFadden, Co-Chair, Joint Audit Committee Members of Joint Audit Committee Annapolis, Maryland Ladies and Gentlemen: We conducted a performance audit to determine the accuracy of certain fiscal year 2000 Managing for Results performance measures reported by the University System of Maryland University of Maryland, College Park in the fiscal year 2002 budget request. We also determined whether adequate control systems were in place over the collection and reporting of related performance measurement data. In accordance with a plan approved by the General Assembly budget committees and the Joint Audit Committee, the performance measures audited were selected by the Department of Legislative Services Office of Policy Analysis. We have developed a system to categorize the results of our audit of performance measures. Measures are designated as either Certified, Certified with Qualification, Factors Prevented Certification or Inaccurate, and are further described in Exhibit 2. Of the five University performance measures we audited, four measures were Certified and the remaining measure was Certified with Qualification. With respect to the measure categorized as Certified with Qualification, we determined that the definition of the measure in the fiscal year 2002 budget submission lacked clarity. We wish to acknowledge the cooperation extended to us during our audit by the University of Maryland, College Park. Respectfully submitted, Bruce A. Myers, CPA Legislative Auditor

4 (This Page Intentionally Left Blank) 2

5 Background Information In July 1997, the Governor s initiative Managing for Results was implemented to increase governmental accountability by utilizing resources more effectively and efficiently to meet the needs of the State s customers and stakeholders (such as the citizens of Maryland). Managing for Results is a future-oriented strategic planning process that establishes direction and priorities for State programs to achieve meaningful results. The success of a program in achieving desired results is determined through the development and monitoring of program performance measures. Managing for Results was implemented over a three-year period beginning with the fiscal year 1999 budget submission and is linked to the budget and decision making process. Specifically, State agencies are required to submit missions, goals, objectives and performance measures for each program to the Department of Budget and Management as part of their annual budget requests. This information may then be considered in determining Statewide spending priorities and the allocation of resources in agency budgets. The General Assembly budget committees and the Joint Audit Committee approved the plan developed by the Department of Legislative Services whereby the Department s Office of Policy Analysis identifies performance measures in the budget requests of selected agencies to be audited by the Office of Legislative Audits. In accordance with this plan, the Office of Policy Analysis identified fiscal year 2000 performance measures at eight agencies for audit. This report contains the results of our audit of the sixth agency and comprises five measures. We have developed a system to categorize the results of our audit of performance measures. Measures are designated as either Certified, Certified with Qualification, Factors Prevented Certification or Inaccurate. The certification categories are defined in Exhibit 2. These categories were assigned based on a combination of the adequacy of controls over a performance measure, consistency between the performance measure calculation method and the performance measure definition, and testing of source documents. 3

6 (This Page Intentionally Left Blank) 4

7 Findings and Recommendations Certification Results for Selected Fiscal Year 2000 Performance Measures Performance Measure (See Exhibit 1) Results Reported 1 Level of Certification (See Exhibit 2) Comments Second-year freshman retention rate all UM students First-time freshman six-year graduation rate all UM students Five-year full-time student graduation rate all full-time UM students Number of degrees awarded in Information Technology Alumni satisfaction with job preparation 90.4% Certified 63.2% Certified 82.0% Certified 715 Certified 89% Certified with Qualification The performance measure was not clearly defined in the fiscal year 2002 budget submission (See Finding #1). 1 Source: Fiscal Year 2002 Budget Submission 5

8 Finding #1 The performance measure alumni satisfaction with job preparation was not clearly defined in the fiscal year 2002 budget submission. Analysis The University did not clearly define the performance measure alumni satisfaction with job preparation in the fiscal year 2002 budget submission. Specifically, the University excluded from its calculation survey responses received from alumni who were only employed part-time even though the survey instructed these alumni to respond to the question concerning their degree of satisfaction with job preparedness. In addition, respondents not currently working, but who indicated they were seeking work were instructed not to respond to the satisfaction in job preparedness question. Finally, the University considered survey responses indicating job preparation to be excellent, good or fair as being satisfied with job preparation. However, the level of satisfaction measured, as well these exclusions, were not clearly disclosed in the fiscal year 2002 budget submission. Recommendation #1 We recommend that the University clarify the definition of alumni satisfaction with job preparation in future budget submissions. 6

9 Scope, Objectives and Methodology Scope We conducted a performance audit of selected fiscal year 2000 performance measure results reported by the University System of Maryland University of Maryland, College Park to determine whether they were accurately reported and whether adequate control systems were in place over the collection and reporting of data used to measure performance. This audit was conducted under the authority of the State Government Article, Section of the Annotated Code of Maryland and was performed in accordance with generally accepted government auditing standards. The General Assembly budget committees and the Joint Audit Committee approved the plan developed by the Department of Legislative Services whereby the Department s Office of Policy Analysis identifies performance measures in the budget requests of selected agencies to be audited by the Office of Legislative Audits. In accordance with this plan, the Office of Policy Analysis identified fiscal year 2000 performance measures at eight agencies for audit. This report covers our audit of the University of Maryland, College Park and comprises five performance measures. We have previously reported the results of our audits of five agencies; the audit results of the remaining two agencies will be included in forthcoming reports. Objectives The objectives of our audit were: (1) to determine whether the fiscal year 2000 measurement results for the selected performance measures have been accurately reported; and (2) to determine whether adequate control systems are in place over the collection and reporting of the data related to the measurement results. Our performance audit did not include an assessment of whether the performance measures reviewed are consistent with the goals and objectives of the related programs or are meaningful indicators of program performance. Methodology To accomplish our objectives, we interviewed University personnel and reviewed performance calculations for accuracy and ensured that these calculations were consistent with the definition of the performance measure as noted in Exhibit 1. We utilized sampling techniques or other methods as deemed appropriate to test the related source documents to verify the accuracy of the reported measure. We also analyzed the University s performance measurement data collection and reporting activities to evaluate whether proper controls were in place. Additionally, we developed a system to categorize the results of our audit of performance measures. The four categories represent varying levels of certification of the accuracy of the fiscal year 2000 performance reported by the University. The categories of performance certification are defined in Exhibit 2. 7

10 Our fieldwork was completed during April and May The University s response to our finding and recommendation appears as an appendix to our report. As prescribed in the State Government Article, Section of the Annotated Code of Maryland, we will advise the University regarding the results of our review of its responses. 8

11 Exhibit 1 Definition of Performance Measures Audited Performance Measure Definition Second-year freshman retention rate all UM students First-time freshman six-year graduation rate all UM students Five-year full-time student graduation rate all full-time UM students Number of degrees awarded in Information Technology Alumni satisfaction with job preparation The measure reflects the percentage of Fall 1999 first-time, full-time, degree-seeking students who were registered for the Fall 2000 semester. The measure reflects the percentage of Fall 1994 first-time, full-time, degree-seeking students who had earned a bachelor s degree at the University by August This measure reflects the percentage of Fall 1994 first-time, full-time, degree-seeking students who had attended the University continuously at a full-time rate and had earned a bachelor s degree by August The measure represents the number of bachelor s, master s and doctoral degrees awarded in Information Technology programs (such as computer science, computer engineering and software engineering) during fiscal year The measure reflects the percentage of alumni surveyed within one year after graduation who were employed full-time and rated their satisfaction with job preparedness as excellent, good or fair. 9

12 Exhibit 2 Categories of Performance Certification Category Certified Certified with Qualification Factors Prevented Certification Inaccurate Definition Reported performance was reasonably accurate. Reported performance was reasonably accurate but either minor deficiencies were noted with the supporting documentation, or controls were not sufficient, or the methodology used to calculate reported performance was not consistent with the measure definition. Actual performance could not be verified as documentation was unavailable and/or controls were not adequate to ensure the accuracy of reported results. Reported performance differed significantly from actual performance. 10

13

14 RESPONSE TO THE LEGISLATIVE AUDIT REPORT UNIVERSITY SYSTEM OF MARYLAND MANAGING FOR RESULTS FISCAL YEAR 2000 PERFORMANCE MEASURES UNIVERSITY OF MARYLAND, COLLEGE PARK AUGUST, 2001 Finding # 1 The performance measure "alumni satisfaction with job preparation" was not clearly defined in the fiscal year 2002 budget submission. The University's Response: In limiting the measure of "alumni satisfaction with job preparation" to those alumni employed fulltime, the University was seeking to remain consistent with prior alumni satisfaction measures, which focused primarily upon those alumni working full-time, and to respond based upon its informal understanding of what the Department of Budget and Management (DBM) sought in the measure. However, the University acknowledges the lack of clarity in the wording of the measure as it was presented, and the possible confusion that could result. In future submissions the University will follow the operational definition agreed to this past Spring (2001) by the University System of Maryland (USM) and DBM, which defines the measure as; "The percentage of bachelor's degree recipients employed full-time within one year of graduation and who rated their education as excellent, good, or adequate (fair) preparation for their job." In addition, the University will include language in the footnotes that highlights the definition of the measure, including the specific delimiters (i.e., bachelor's degree recipients employed full-time within one year of graduation who rated their education as excellent, good, or adequate (fair) preparation for their job). Per the USM-DBM negotiated definition referenced above, the University also will continue to exclude from the measure the responses of those alumni who are not employed full-time. However, this exception will be highlighted in the footnote. On the issue of excluding from its calculations responses from surveys that could not be positively identified, the University believes that this is sound survey practice, in that it ensures the response of each alumnus will be counted only once. To do otherwise would risk contaminating the data. However, the University agrees to continue to monitor responses from the "non-identified respondents" in order to ascertain whether satisfaction rates for this fairly small group appear to significantly diverge from rates for identified respondents. If this occurs, the University will alert USM and MHEC of its findings and discuss with them a plan for follow-up studies should such studies be necessary. Finally, per the agreed upon definition cited above, the University will continue to combine the alumni ratings of "excellent", "good", or "adequate (fair)" in order to obtain the required single measure of employment preparation satisfaction. However, the fact that this measure represents an aggregate of survey responses (excellent, good, or fair) will be clearly noted in the footnote.

15 RESPONSE TO THE LEGISLATIVE AUDIT REPORT UNIVERSITY SYSTEM OF MARYLAND MANAGING FOR RESULTS FISCAL YEAR 2000 PERFORMANCE MEASURES UNIVERSITY OF MARYLAND, COLLEGE PARK AUGUST, 2001 Actions to be taken: Per the instructions of USM and DBM, the University will continue to calculate satisfaction measures based upon "bachelor's degree recipients employed full-time within one year of graduation and who rated their education as excellent, good, or adequate (fair) preparation for their job." However, it agrees to highlight this definition prominently in the footnotes of the MFR, a process that will now be easier given DBM's lifting of the space limitations. The University also will monitor the satisfaction levels expressed in those excluded surveys and alert USM and MHEC if any major discrepancies occur between them and the other (identified) surveys. Finally, the University will continue to combine alumni ratings of "excellent", "good", or "fair" into one overall rating of alumni satisfaction with job preparation; however, this aggregation will be clearly explained in the footnotes of the MFR document. Summary: The University of Maryland is pleased with the findings of the Office of Legislative Audits analysis. During our debriefing meeting at the conclusion of the process, our auditor stated that our processes were "spot-on," and that they almost never find agencies with policies and procedures in place as extensive and reliable as they found at the University of Maryland, College Park. We will continue to attend to the MFR process as we have, and will make the changes in our definitions as stated above, to be in full compliance with the auditor's recommendations.