Report of the Auditor General of Québec to the National Assembly for

Transcription

1 Report of the Auditor General of Québec to the National Assembly for Value-for-Money Audit Highlights Fall 2011

2 Cover photo Parliament Building, Daniel Lessard, National Assembly Collection

3 Report of the Auditor General of Québec to the National Assembly for Value-for-Money Audit Highlights Fall 2011

4

5 Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 5 Table of contents Observations of the Auditor General, Mr. Renaud Lachance Government Actions to Combat Poverty and Social Exclusion Food Services in the Health and Social Services Network Collection of Tax Debts Educational Childcare Services: Quality, Performance and Reporting Chapter 6 Société de l assurance automobile du Québec : Compensation of Accident Victims Chapter 7 Aide financière aux études

6

7 Highlights Observations of the Auditor General, Mr. Renaud Lachance 1 Introduction The Québec National Assembly has entrusted the Auditor General with the mandate of facilitating, through audit, parliamentary control over public funds and other public property. This mandate comprises, to the extent deemed appropriate by the Auditor General, financial audits, audits to ensure the compliance of operations with statutes, regulations, policy statements and directives, as well as value-for-money audits. The Auditor General s fields of jurisdiction mainly encompasses the government, its agencies and its corporations; the Auditor General is also empowered to audit funds paid in the form of subsidies. In the annual report that the Auditor General submits to the National Assembly, he draws attention to any topic arising from his work that deserves to be brought to the attention of parliamentarians. This document is published in several volumes. The current volume is devoted to the value-for-money audits carried out over the last 12 months and contains the related findings, conclusions and recommendations. This volume begins with a chapter that gives the Auditor General the opportunity to establish a more personal contact with the reader. In this chapter, the Auditor General comments on the work that he has carried out in recent years in the area of public infrastructures. He then presents his main questions in that regard so as to contribute to the reflections of managers and parliamentarians who have an interest in the subject. Infrastructure Public infrastructures are important, both in number and in the enormous investments that they require. In Québec, as elsewhere in the world, a large number of infrastructures were put into place between 1950 and As time passed, the maintenance, repair and replacement of those infrastructures were not always adequately carried out. In 2007, the government found that several of them were in an advanced state of obsolescence and had to be renovated or replaced. At that time, it estimated the maintenance deficit for its infrastructures at more than $27 billion. 1 As part of the effort to make up for this, the Québec government is responsible for $20.7 billion while its partners (mainly the federal government and the municipalities) are responsible for $6.3 billion. That same year, to better frame the management of its infrastructures, the government passed An Act to promote the maintenance and renewal of public infrastructures and launched the Québec Infrastructure Plan, which is aimed at modernizing, repairing and preserving public infrastructures. Following that, in 2008, a framework policy on the governance of major infrastructure projects was developed. 1. This information is taken from the Québec infrastructure plan entitled Foundations for Success. Value-for-Money Audit Fall

8 Chapter In 2011, the Québec Infrastructure Plan was adopted. It provides for investments of $44.6 billion, that is, $28.6 billion for asset rehabilitation, including the elimination of a portion of the accumulated maintenance deficit ($7 billion), $11.2 billion for infrastructure improvement and replacement and $4.8 billion for the completion of projects begun before (Table 1). Table 1 Investments in the Infrastructure Plan (in millions of dollars) Sector Asset rehabilitation Improvements and replacement Completion* Total Road network 11, , , ,860.9 Public transit 1, ,953.1 Health and social services 5, , ,074.4 Education 5, ,254.2 Culture ,142.9 Municipal infrastructures 2, ,869.8 Public housing ,064.2 Justice and public safety ,022.7 Others , ,313.6 Total 28, , , ,555.8 Percentage 64% 25% 11% 100% * These amounts are earmarked for projects that begun before Source: Expenditure Budget. Previous Audits 1.8 Since becoming Auditor General, I have been interested in the management of public infrastructures. In 2006, I set for myself a strategic planning objective to inform parliamentarians on the advancement of major infrastructure projects. Since 2004, the following audit missions have included the management of capital projects and public infrastructure projects: Project to extend the subway network on the territory of the city of Laval (December 2004); Water services and sustainability of infrastructures (June 2005); Management of real estate projects (Ministère de la Santé et des Services sociaux and Corporation d hébergement du Québec) (June 2006); Société immobilière du Québec (March 2008); Special audit carried out at the Université du Québec à Montréal. Part II Main factors responsible for the losses of the Université du Québec à Montréal (June 2008); 2 Report of the Auditor General of Québec to the National Assembly for

9 Highlights Watches over major construction projects: Projects to modernize Montréal s University Health Centers (December 2006, April 2009, November 2009, June 2010); Implementation of the integration resource management business solution (SAGIR) project (December 2006); Dossier de santé du Québec (May 2009, May 2010, May 2011); Business Projects Enabled by Information Resources and Government Supervision (May 2011) Those missions made it possible for me to identify several weaknesses, which are often repeated from one department or body to another. Most of the projects examined had shortcomings in the planning process at every stage of their development. The governance structure in place did not necessarily facilitate the success of the projects. For some projects, there was no project manager responsible for the results obtained from the beginning to the end of the work carried out. Also, the role and duties of all involved in the decision-making process were not always well defined. Decision makers are often given an incorrect or imprecise picture because of ill-defined needs, poor rationales, analyses lacking rigour and unrealistic or incomplete cost estimates. For example, in the case of the modernization of Montréal s University Health Centers, the documents used to support decision-making do not have a section covering consideration of the scope of the work, capital budgets and operating budgets needed to operate the infrastructures. I also observed the great difficulty that public bodies have in integrating the scope of work, costs and deadlines and in meeting these three important criteria. As a result, public bodies face numerous problems in delivering what they initially planned. Furthermore, those responsible for governance did not always monitor projects with the required vigilance. In fact, project summaries are often non-existent and reporting is inadequate. In other respects, several departments and bodies do not have a complete picture of the state of their infrastructures, which does not favour the development of the planning of maintenance, repair and renovation work. Value-for-Money Audit Fall

10 Chapter 1 Other Administrations Legislative auditors in the rest of Canada and in other countries are also interested in infrastructure management. During the last three years, auditors of Canada, Great Britain, Ireland and Australia have produced 24 reports mainly covering this subject. Their comments on infrastructure management are in line with my own. Planning is the most frequently studied phase. Most of the reports (21 out of 24) concern governance and the processes for acquisitions and approvals. The project monitoring (15 reports) and the management of public infrastructure maintenance and repairs (12) are the other subjects often examined. I observed with interest that some administrations have implemented systematic monitoring mechanisms for major infrastructure projects. For example, in three of the four legislative assemblies of Australia, specialized parliamentary committees carry out studies on major infrastructure projects. Each committee studies the projects that meet certain conditions and monitors them. The federal parliament of Australia has the most fully developed model; it must decide on a project s usefulness, needs, cost-efficiency ratio of the proposal, projected revenues if any and the project s present and future cost. Important Questions Because Québec s parliamentarians are interested in major infrastructure projects, I carried out an analysis to determine the Auditor General s future involvement in this area. My analysis emphasizes a problem that is specific to each phase in carrying out a project, whether or not the results of the phase contribute to the success of the project. In the following paragraphs, the reader will find a brief summary of a project s phases and the questions related to them. These questions are at the heart of my concerns. I am raising them to give food for thought to managers and parliamentarians who are interested in the sound management of public infrastructure projects. Schedule 1 gives examples of sound practices associated with these important questions. Setting Priorities 1.19 Each year, the government injects large sums to eliminate the maintenance deficit, to repair existing infrastructures and to carry out new projects. All the needs cannot be met at the same time. That means that departments and bodies must be acquainted with their needs and the state of their infrastructures and must identify the projects to be carried out on a priority basis. I expect them to answer the following questions in the affirmative: Has a master plan for construction been developed and have the criteria for classifying and selecting projects in an objective manner been set and made known? Has the process for setting the priorities for choosing projects been rigorously followed? 4 Report of the Auditor General of Québec to the National Assembly for

11 Highlights Strategic Analysis 1.20 By means of a structured presentation, a strategic analysis is intended to show the relevance of carrying out a project. At a minimum, it must describe the needs and the anticipated results, analyze the various means to meet needs, the order of magnitude of the costs and main stakes. The relevant questions for the strategic analysis include the following: Does the infrastructure project provide a solution for a well-defined problem? As presented, is the infrastructure project the best way to meet the needs? At this stage, is the analysis sufficient to make a decision? Project Planning 1.21 This phase s objective of departments and bodies is to choose the best option to adequately meet the needs and to agree on the way in which the project will be carried out, which then allow them to be in a position to undertake the acquisition process that will lead to the choice of a supplier for the execution of the project. Several elements must be considered. Governance. The effectiveness and independence of the people who assure governance are conditions for the success of the projects. It is essential that the roles and responsibilities be clearly determined to facilitate the best possible reporting. In that regard, I have the following questions: Does the governance make it possible to specify the responsibilities of all stakeholders involved? Do the people who ensure governance have the necessary independence? Does the organization have in place the resources needed to manage the planning steps? Choosing an option. The purpose of this phase is to determine the best solution for meeting the needs. A detailed assessment of each option possible (for example, construction, renovation, leasing, changes in approach) will, at a minimum, have the following elements: technical and technological feasibility, human resource factors, regulatory impacts, consideration of the principles related to sustainable development, risk analysis, financial and sensitivity analysis. For the option chosen, preparatory studies must clearly define the requirements related to the infrastructure s functional, operational and physical organization and the quality objectives. They must answer the following questions: Are the initial project needs and the anticipated results still the same as those determined by the strategic analysis? Did the department or the body study various options for solving the problems that led to the project? Was the option selected by the department or the body subjected to rigorous and detailed studies? Value-for-Money Audit Fall

12 Chapter 1 Choosing an implementation method. Once the project has been defined and a construction decision has been made, the most appropriate implementation method must be chosen. The main distinction between the various methods is the allocation of risks and the division of responsibilities. Regardless of the project s size, the choice must be backed up by analyses. I expect the analyses to take the following aspects into account: For the option selected, were several implementation methods carefully analyzed? Do plausible assumptions and reliable estimates support the quantitative analysis (cost estimates and operational budgets required to operate the infrastructure)? Were objective factors used for the qualitative analysis? Is the information on which decision makers will base their approval of the implementation method complete, reliable and sufficient? Acquisition process and final approval. For turnkey and public-private partnership methods of implementation, approval of the initial business plan by decision makers triggers the acquisition process. After developing performance specifications, the department or body can issue an invitation for proposals and then reach an agreement with the chosen partner. In the case of a traditional method or management contracting, the decision makers first authorize the development of preliminary plans and specifications. Then, detailed plans and specifications are produced and approved. Finally, a call for tenders is issued, and the contract with the chosen tenderer is signed. My questions on the acquisition process are the following: Is the documentation provided to the decision makers for selecting the supplier complete, precise and reliable, and does it contain the recommendations of the various stakeholders? Is the documentation produced for the invitation for proposals or the call for tenders complete, precise and in conformity with the authorizations obtained? Does the acquisition process in effect make it possible to select the best offer? For major projects, is the information provided to the Cabinet for approval of the agreement with the selected partner relevant and complete? Project Execution 1.22 This phase concerns the actual construction; the suppler is selected and the contract is signed. The project manager is responsible for analyzing change requests and their impact on the work, its quality, its costs and the project timeline. All change requests must follow an established authorization process. My questions on project execution are the following: Does a clearly designated project manager who is recognized by all stakeholders rigorously and continuously monitor the steps associated with the project s execution? Is there a control process for project changes made as execution proceeds? Are there assurances that such changes will not have negative repercussions on the project? 6 Report of the Auditor General of Québec to the National Assembly for

13 Highlights Putting the Infrastructure Into Operation and Monitoring 1.23 Once the infrastructure has been completed, the entity must proceed with the final acceptance of the construction work. It must also have planned the transition and the start-up of all the services needed to operate the new infrastructure. Furthermore, systematic monitoring throughout the duration of the contract must be carried out to ensure that the entity does not assume responsibilities or risks that have been assigned to the supplier. My questions in this regard are the following: When the project has been fully completed, are there assurances that the project parameters correspond to those in the initial agreement? Will a watch be carried out during the transition period? Has a plan been made for putting the infrastructure into operation? Have follw-up been made for invoice monitoring throughout the project s life cycle (including maintenance and operation)? Managing Contracts for Professional Services 1.24 Infrastructure projects require expert knowledge at certain steps of planning and implementation. Various aspects related to professional service contracts must be monitored: guidelines for the acquisition process for professional services, determination of the need for expertise, contracts awarding and follow-up. My questions on contract management include the following: Has it been determined that internal resources are not able to provide the services that will be requested from external resources? Have the rules for awarding professional service contracts are followed? Has the price-quality ratio been included in the evaluation of professional service contracts? For negotiated contracts by mutual agreement, are there assurances that the service supplier has the required competencies? To what extent do the services received meet the needs? Management of the Maintenance and Repair of Public Infrastructures The purpose of the 1.25 Act to promote the maintenance and renewal of public infrastructures is to ensure that state investments in public infrastructures are transparent and made in accordance with best management practices, and that they are properly apportioned between infrastructure maintenance and infrastructure development. The obligation to eliminate the maintenance deficit accumulated over the preceding years is clearly specified, and the elimination must be completed within 15 years. Value-for-Money Audit Fall

14 Chapter The repair needs must be determined on the basis of recognized standards and the characteristics of the infrastructures (age, operating life and replacement cost). The maintenance deficit must be correctly estimated, and investments must be made as scheduled. Reporting must make it possible to assess the performance of public bodies in this respect. Sound management of this activity requires affirmative answers to the following questions: Is the deficit estimate up-to-date and based on a recognized methodology? Have the sums invested made it possible to reduce the maintenance deficit since 2007? Have management frameworks been designed so as to respect recognized standards for the maintenance and repair of assets and so that sustained investments can be made for infrastructure maintenance? 8 Report of the Auditor General of Québec to the National Assembly for

15 Highlights Schedule 1 Important Questions and Sound Practices Setting Priorities Has a master plan for construction been developed and have the criteria for classifying and selecting projects in an objective manner been set and made known? Has the process for setting the priorities for choosing projects been rigorously followed? Strategic Analysis Does the infrastructure project provide a solution for well-defined problem? As presented, is the infrastructure project the best way to meet the needs? At this stage, are the analyses sufficient to make a decision? Project Planning Governance Does the governance make it possible to specify the responsibilities of all stakeholders involved? Do the people who ensure governance have the necessary independence? Does the organization have put in place the resources needed to manage the planning steps? Choosing an option Are the initial project needs and the anticipated results still the same as those determined by the strategic analysis? Did the department or body study various options for solving the problems that led to the project? Master plan for construction in line with the strategic orientations Mechanism to set priorities and make criteria known (for example, safety, obsolescence, technical feasibility, etc.) Tool for comparing projects and setting priorities based on the stakes of each project Approval of the criteria for setting project priorities Respecting the criteria Documents to support decisions Project s conformity with orientations Needs identified and stated Anticipated results clearly determined Relevant documentation Examination of possible options Advantages and disadvantages of the various options and presentation of the selected option Methodology for determining the order of magnitude of capital costs Plan for managing sociopolitical risks (presenting the stakes, clientele, environmental factors, etc.) Plan for managing communications (announcement, timeline, etc.) Cost estimation methodology supported by the required studies and review of reasonableness by means of comparisons Authorization of budget availability Delegation of authority and powers clearly defined Responsibilities of each involved party defined and interpreted uniformly for all mechanisms for decision-making (information, timeliness, decision follow-up) Comparability with the governance of similar projects Form a steering committee at the start of the project and a project office headed by a project manager; skills and experience of the office s resources Availability and skills of the resources to carry out the project s planning studies Update project content and results based on changes since the strategic analysis was made Documents monitoring changes that take into account the needs and anticipated results Rigorous and documented analyses of feasible options for construction and other elements, in accordance with the Guide d élaboration du dossier d affaires (projects over $40 million) or the reference guidelines of the entity (projects under $40 million) Value-for-Money Audit Fall

16 Chapter 1 Schedule 1 Important Questions and Sound Practices (continued) Was the option selected by the department or body subjected to rigorous, detailed studies? Is the business case (initial or other) complete and exempt of contradictions, errors, omissions or inconsistencies that could have an influence on its conclusions? Have the sociopolitical stakes been identified and updated? Choosing an Implementation Method For the option selected, are several implementation methods rigorously analyzed? Do plausible assumptions and reliable estimates support the quantitative analysis (cost estimates and operational budgets required to operate the infrastructure)? Were objective factors used for the qualitative analysis? Where relevant, have the recommendations arising from the diligent review been followed? Is the information on which decision makers will base their approval of the implementation method complete, reliable and sufficient? Justification of the selected option based on needs and anticipated results Preparatory studies (performance specifications, operational and fonctional program or other) covering technical requirements, performance requirements and the risks inherent in the selected option Estimate of the net costs of implementing the project and documentation of sources (for example, comparative data and standards in effect in the marketplace) Compliance with the directives in the Guide d élaboration du dossier d affaires Analyses and recommendations made by the Secrétariat du Conseil du trésor Analyses and recommendations made by the due diligence committee Corrections made to the business case to implement recommendations Plan for managing the sociopolitical stakes (main allies and opponents and their reasons) Communication plan Risk reduction measures and estimates of their costs Relevance of the implementation methods analyzed Criteria for assessing the potential of the implementation methods (for example, for a public-private partnership, present the following characteristics: separability and measurability and existence of a competitive marketplace) Rigorous, comparative analysis of the implementation methods (quantitative and qualitative) Main assumptions that are realistic and comparable to those for similar projects (discount rate, risk reserve, transferable risks, private sector s efficiency rate, rate of return, financing rate, etc.) Realistic estimates of the costs and budgets needed to operate the infrastructure Validation of estimates by independent experts (cost assumptions and estimates) Scope of the analyses (life cycle, if relevant) Sensitivity analyses (impact of variations of one or several combined assumptions) Relevant qualitative factors that are comparable to those used for other similar projects Independence and expertise of the members of the due diligence committee Committee s examination of the analysis supporting the proposed method (estimates, assumptions, comparable projects, etc.) Adjustments to the analyses and conclusions if needed after receiving the committee s recommendations Brief submitted to Cabinet that includes the strategic planning file the initial business case recommending the method to be selected the analyses and recommendations of the Secrétariat du Conseil du trésor and the due diligence committee Order in Council attesting approval 10 Report of the Auditor General of Québec to the National Assembly for

17 Highlights Schedule 1 Important Questions and Sound Practices (continued) Acquisition Process and Final Approval Is the documentation provided to the decision makers for selecting the supplier complete, precise and reliable, and does it contain the recommendations of the various stakeholders? Is the documentation produced for the invitation for proposals or the call for tenders complete, precise and in conformity with the approvals obtained? Does the acquisition process in effect make it possible to select the best offer? For major projects, is the information provided to Cabinet for approval of the agreement with the selected partner relevant and complete? Project Execution Does a clearly designated project manager who is recognized by all those involved stakeholders rigorously continuously monitors the steps associated with the project s execution? Is there a control process for project changes made as execution proceeds? Are there assurances that such changes will not have negative repercussions on the project? Has a process been established for disputes arising during the execution of the project? Integration of the main conclusions of the project cost analysis Mechanisms to ensure that information provided is complete and specific Analyses and approvals of the final business case by Infrastructure Québec, the due diligence committee and the Secrétariat du Conseil du trésor Integration of the recommendations into the final business case Justification for recommendations that are not applied Criteria defined and approved during previous stages in the documentation and approval of any subsequent changes Cost estimates that respect the approved budgets Independence and expertise of the members of the selection committee Absence of any conflict of interest or appearance of conflict Analysis of suppliers financial soundness Acquisition process: measures taken to ensure conformity and fairness in the handling of invitations for proposals and calls for tenders Assessment grid comparable to those used for other projects, especially with regard to weighting; weight given to quality Analysis table for the various offers or proposals Analysis of differences between the proposals and cost estimates, if relevant Analysis of the differences between the proposal selected and the project approved during the invitation for proposals or call for tenders Identification of contract terms and conditions that are not standard, analysis of their impact and their acceptability Analysis of changes in risk sharing with regard to what was anticipated when the invitation for proposals or call for tenders was made Presence of a project office headed by a project manager: definition of roles and responsibilities, their acceptance and dissemination Decision-making mechanism (timing, continuous monitoring of the execution of decisions) Project management plan (project contents, timeline, costs, anticipated quality, risks) Mechanisms for follow-up on the management plan and systematic monitoring of the costs, timeline, risks and the advancement of the work Follow-up on the terms and conditions for payments Process for change requests Authorization level required for changes Analysis of change requests (relevance, offset measure) Analysis of impact on the project (costs, timeline, quality, scope) Presence of a dispute resolution mechanism and its application Nature of events resulting in disputes Impact of disputes on the timeline, costs, scope and quality Value-for-Money Audit Fall

18 Chapter 1 Schedule 1 Important Questions and Sound Practices (continued) Putting the Infrastructure Into Operation and Monitoring When the project has been fully Mechanisms to ensure that the project meets the original needs completed, are there assurances that the project parameters correspond and the stipulations in the agreement Delivery of as-built plans: analysis of the impact on subsequent to those in the initial agreement? maintenance and repairs Notice of the work s conformity (provisional and final acceptance) Compliance with contract stipulations before making payments Will a watch be carried out during the Presence of a plan for putting the infrastructure into operation transition period? Has a plan been Actions or control points planned for putting the project into operation made for putting the infrastructure into operation? (frequency, period) Follow-up on the contractual terms and conditions for payments Have follow-up been made for invoice Process for follow-up on declarations and invoicing monitoring throughout the project s life Follow-up on penalties and fines cycle (including maintenance and operation)? Follow-up on payment conditions (audit on the anticipated results) Managing Contracts for Professional Services Has it been determined that internal Recognition of the competencies, expertise and availability of the entity s resources are not able to provide the services that will be requested internal resources No repetitive contract renewals from external resources? Analysis of the impact on internal expertise Have the rules for awarding professional Rules for the type of contract service contracts been followed? Respect for approval levels Use of exceptions for public contracts Possibility for contract splitting and impact on the method for receiving submissions Has the price-quality ratio been included Considering quality as an assessment criteria in the evaluation of professional service Weighting quality in relation to the adjusted price contracts? For negotiated contracts by mutual Survey of the past experience of the selected consultants (solicitation method) agreement, are there assurances that the service supplier has the required competencies? Are the professional service contracts Organization s capacity to supervise professional service contracts awarded during the planning phase Description of deliverables managed properly? Risk management related to contracts To what extent do services received meet Use of incentives to improve contract performance the needs? Follow-up on the use of deliverables Follow-up on payments vs. invoicing and the deliverables Ratio of the costs of professional services to project costs, with respect to standards of projects of the same size and nature 12 Report of the Auditor General of Québec to the National Assembly for

19 Highlights Schedule 1 Important Questions and Sound Practices (continued) Management of the Maintenance and Repair of Public Infrastructures Is the deficit estimate up-to-date and Quality of the methods and data used to estimate the deficit based on a recognized methodology? and the elimination period Results of infrastructure inspections Estimates based on certain factors (for example, area) Reasons for deficit Have the sums invested made it possible Inventory of all infrastructures under the responsibility of each department to reduce the maintenance deficit since 2007? and body Investment variations in this regard since 2007 Follow-up on the reduction of the maintenance deficit by departments and bodies Follow-up system for departments and bodies Accountability reports Have management frameworks been Basis of reference for developing the management frameworks designed so as to respect recognized Revision of the management frameworks standards for the maintenance and repair of assets and so that sustained investments can be made for infrastructure maintenance? Respect for management frameworks Value-for-Money Audit Fall

20

21 Highlights Government Actions to Combat Poverty and Social Exclusion 2 Audit Objectives Our work focused on the National Strategy to Combat Poverty and Social Exclusion, published in 2002, and on the two government action plans based on it. The first action plan ( ) proposed 47 measures totalling investments nearly $4.5 billion. The second action plan ( ) proposes 70 measures totalling investments nearly $7 billion. The specific purpose of our audit was to obtain the assurance that the Ministère de l Emploi et de la Solidarité sociale (MESS) coordinates the implementation of these government plans by setting up, in collaboration with its partners, management mechanisms that make it possible to carry out and effectively follow up on actions and concretely assess results with a view to making sustainable progress. Audit Results We present here the principal findings of our audit concerning government actions to combat poverty and social exclusion. The MESS did not give any directives to government departments and agencies on measures to be proposed for the action plans. It did not formulate specific criteria to guide them and did not require targets and deadlines for the proposed measures. Consequently, the plans adopted contain a large number of measures that do not always have the desired level of harmonization. The MESS did not specify how to determine the sums to be included in the calculation of the investments announced in the action plans. In fact, it did not ensure that the sums considered were calculated on the same basis, which would have facilitated harmonization of the various investments. Consequently, it is not possible to have an accurate picture of the situation. The annual activity reports, which are prescribed by law and used to report on the government action plan, are incomplete and lack clarity. They make it difficult to measure the progress made with respect to measures and estimate what remains to be done. The MESS did not adequately plan the follow-up on the measures in the action plans and the assessment of the results. The main measures, in terms of investment size, were assessed, but overall, the assessment efforts are incomplete and were not made in a timely manner. Several deadlines prescribed by law for the submission of reports and notices were not met. Government planning efforts should have been carried out in a more rigorous application framework, although the mechanisms that were put into place may contribute to awareness and consensus building among the various stakeholders. We found that the foundation for a follow-up on measures and especially for an assessment of results is insufficient. The Report is available at Value-for-Money Audit Fall

22 Chapter 2 Recommendations The Auditor General has made recommendations to the MESS. All of them are shown to the right. The audited entity had the opportunity to comment on these recommendations; its comments can be found in paragraph We want to point out that it accepted all of the recommendations. Recommendations to the Department 1 Set specific criteria for choosing the measures to be included in a future action plan related to the National Strategy to Combat Poverty and Social Exclusion. 2 Ensure that targets and deadlines are set for the proposed measures. 3 Implement the management mechanisms needed so that for a future action plan, the investments considered are harmonized with each other and with the plan s various measures. 4 Ensure that a follow-up on the measures included in the action plan and a regular assessment of the results associated with the measures are made. 5 Make better reporting by improving the contents of the annual activity report. 6 Take the steps needed to ensure that the deadlines prescribed by law are met. 16 Report of the Auditor General of Québec to the National Assembly for

23 Highlights Food Services in the Health and Social Services Network 3 Audit Objectives Our audit s objectives were to ensure that: the Ministère de la Santé et des Services sociaux (MSSS) and the audited agencies and institutions assume their roles and responsibilities regarding services relating to food, and carry out a performance assessment of these services; the audited institutions manage their services concerning food in a manner that promotes an economic and efficient use of resources as well as protect the environment, in accordance with the regulations; the audited institutions implement clinical nutrition measures that facilitate health maintenance or health recovery and improve the well-being of care centres residents. The Report is available at Audit Results We present here the principal findings of our audit concerning food services in the health and social services network. Few analyses of food service management and production methods are made by the three audited centres de la santé et des services sociaux (CSSS). Thus, they do not know whether the methods in use are effective. In fact, no major changes have been observed in management and in production since 2004, year when the institutions merged into CSSS. A large number of activities and functions cumulated in different service points of one single institution have not yet been integrated. Large variation in the average cost of a meal between Québec institutions and between the audited CSSS: causes of the differences observed not sufficiently analyzed. In , the average cost of a meal varied between $3.92 and $13.79 from one institution to another in the province, but the variations are not explained. For the same year, the unit cost in the three audited CSSS was higher than the average in comparable institutions. The difference is slight for one of them, but much greater for the other two. Hygiene and food safety standards: the risk level for the three audited CSSS is acceptable. However, none of the CSSS has a systematic control mechanism, such as a quality assurance program for hygiene and food safety rules. No assurance that, in residential care centres, users suffering from undernutrition are diagnosed and treated rapidly. The CSSS do not have any formal procedure to ensure that users suffering from undernutrition are systematically identified upon arrival at the residential care centre and periodically thereafter. MSSS and two of the audited agencies: no leadership related to food production in the institutions. These entities do not have a picture of the operating mode of food services, nor have they adopted any position on the procedures to favour. Positive effects of actions taken by the MSSS and the audited agencies on nutritional quality in the network. Such actions should encourage institutions to increasingly offer high-quality food to their clientele. Value-for-Money Audit Fall

24 Chapter 3 Recommendations The Auditor General has made recommendations to the MSSS and the audited agencies and institutions. Most of them are shown to the right. The audited entities had the opportunity to comment on these recommendations; their comments can be found in paragraphs 3.83 to We want to point out that they accepted all of the recommendations. Recommendations to the audited institutions 1 Periodically review their food service management and production methods and take appropriate measures to ensure that they are effective. 2 Regularly carry out analyses of their financial performance and their employee s productivity so as to identify the causes of the differences between the performance of their different service points and the differences between their performance and those of other institutions. 3 Improve the practices related to standardized recipes. 4 Maximize acquisitions from purchasing groups. 6 Set the selling price of meals by taking into account the overall cost (except for the price set in the collective agreement for unionized employees). 7 Increase the computerization of their food services activities and functions and establish links between the various systems. 8 Implement systematic mechanisms to monitor the main elements regarding hygiene and food safety. 9 Ensure that the users in residential care centres who have undernutrition problems are rapidly identified and treated. 11 Ensure that the residential care centres offer professional clinical nutrition services that meet residents care centres needs. 13 Ensure that a variety of foods are offered in residential care centres and review menus regularly. 14 Assure systematic control of the quality of the food served and compliance with therapeutic indications. Recommendation to the MSSS 16 Take measures to ensure that the institutions optimize resources provided to food and clinical nutrition services and that they adopt a plan to carry out the changes deemed necessary. Recommendations to the MSSS and the audited agencies 17 Monitor the institutions performance in regard to food and clinical nutrition services and take measures to motivate them to make improve ments in this regard. 18 Provide support to the institutions and ensure their coordination in terms of food and clinical nutrition services. 18 Report of the Auditor General of Québec to the National Assembly for

25 Highlights Collection of Tax Debts 4 Audit Objectives The Centre de perception fiscale (CPF), a Revenu Québec unit, is responsible for the collection of tax debts. In , the CPF collected $2.3 billion. As at March 31, 2011, the tax debts entered in the Système de perception intégré des créances amounted to $7.6 billion. The purpose of our audit was to obtain the assurance that Revenu Québec: Has an adequate strategy for collecting tax debts; Manages collection files effectively, efficiently and fairly; Assesses the performance of activities related to the collection of tax debts and makes accountability reports. The Report is available at Audit Results We present here the principal findings of our audit concerning the collection of tax debts. The collection strategy in place at Revenu Québec for managing tax debts needs adjustments. Since 2008, tax debts have continued to increase, particularly those of businesses and those aged from two to five years. An increasing number of regular files are awaiting processing by a collection agent, which raises questions on the strategy used to assign files. As at March 31, 2011, there were more than 22,000 unassigned files and more than 16,000 files awaiting re-assignment. From 2008 to 2011, files unassigned for more than three months increased from 28% to 41%. In the case of files awaiting re-assignment, one-third had been in that situation for more than four months. We anticipate collection risks for the debts related to those files and in terms of fairness, because debtors are not subjected to collection actions carried out by an agent during that time. Corrections must be made to the processing of regular collection files. The written communications sent to debtors and third parties are, in some cases, hard to understand or not explicit enough. Furthermore, some important documents are not sent to them. In addition, taxpayers are not well informed on the possibility of requesting a cancellation of penalties and interest when justified by their situation. They should be better informed in this regard. Deficiencies were found with respect to the interventions made to collect debts. For example, the agents knowledge of debtors financial profiles is often insufficient, because some forms are not always filled out, such as the financial analysis, which was missing in 41% of the files where it was required. Among other deficiencies, there were payment arrangements made on the basis of incomplete information or recovery efforts that were carried out with more or less intensity by collection agents. Different actions were observed for similar situations, which puts fairness at risk. Finally, quality control measures need to be strengthened. There are significant differences in the productivity of the regional branch offices. The CPF has not conducted any in-depth analysis of these differences. Such analysis would make it possible to identify situations where productivity gains can be achieved and to highlight management practices that should be shared. Thus, it was more difficult to assess performance in this area. Value-for-Money Audit Fall

26 Chapter 4 Recommendations The Auditor General has made recommendations to Revenu Québec. All of them are shown to the right. The audited entity had the opportunity to comment on these recommendations; its comments can be found in paragraph We want to point out that it accepted all of the recommendations. Recommendations to Revenu Québec 1 Adapt its collection strategy so as to manage all tax debts and meet new challenges in that regard, with a concern for effectiveness, efficiency and fairness. 2 Implement the measures needed so that the collection procedure is carried out well, that it has the expected results, particularly in terms of fairness, and that it is revised as needed. 3 Review communications so that they are clear and understandable for taxpayers, third parties and mandataries and send them in a timely manner. 4 Implement an effective quality assurance process for all its collection activities. 5 Ensure that control points are appropriate and that they meet users needs. 6 Carry out regular follow-ups on the most pertinent indicators so as to assess and compare the performance of the regional branch offices and activity sectors. 7 Analyze the reasons for differences observed in the follow-ups on indicators and apply corrective measures that arise from the analysis. 8 Improve the accountability reporting included in the annual man agement report with respect to its tax debt collection activities so that its performance can be better assessed. 20 Report of the Auditor General of Québec to the National Assembly for

27 Highlights Educational Childcare Services: Quality, Performance and Reporting 5 Audit Objectives The network of childcare centres, subsidized day care centres and home childcare services required a government investment of $2.1 billion to finance the 215,000 reducedcontribution spaces available in The purpose of our audit was twofold. We wished to obtain the assurance that the Ministère de la Famille et des Aînés (MFA): Had taken measures to set up a network of educational childcare services, in accordance with the Educational Childcare Act and sound management rules and practices, and that it had done so efficiently and effectively; Evaluates its performance and makes adequate accountability reports on the quality of educational childcare services. The Report is available at Audit Results We present here the principal findings of our audit concerning educational childcare services. The process for allocating 18,000 new reduced-contribution spaces that was begun by the MFA in 2008 did not make it possible to allocate spaces to high-quality projects, as defined by the MFA. The call for projects was limited to only one month, which is a very short time compared with the four months provided for the current process. When the call for projects was made, the eligibility criteria had not been defined. Furthermore, the analysis was made without an overall assessment of the projects and without setting a pass mark, which would have made it possible to reject projects that did not meet the minimum desired requirements. Several of the selected projects received unsatisfactory assessments for quality (74% of them) and feasibility (19%), based on the MFA s own criteria. Moreover, 29% of the selected projects (3,864 spaces) were located in municipalities where the MFA estimated there was a surplus of spaces. At least 42% of those spaces (1,626) were in new facilities of more than 60 spaces. There was considerable subjectivity in the process. Consequently, it is difficult for the MFA to show that the allocated spaces meet the specific needs that it had identified and its other priorities. The measures proposed by the MFA to improve the quality of childcare services did not receive the desired level of support from childcare services or were not assessed on the basis of results. Only 23% of the childcare services sent the MFA a signed quality-commitment declaration with respect to the structuring measures set out in the continuous improvement plan published in Seven years later, the accreditation process is still only at the project stage and its implementation has been delayed. Furthermore, the MFA makes few efforts to ensure that the home childcare coordinating offices adequately carry out their duties to monitor the quality of the services offered by home childcare providers. In 2011, the MFA tripled the number of inspectors. However, the development of the procedures and tools needed to carry out inspection tasks has been delayed. In the last five years, the MFA has renewed 29% of the permits without first carrying out the inspection provided for in the MFA directive. Finally, the MFA has not implemented any quality assurance mechanism that would make it possible to verify whether inspections are carried out in accordance with the directives. The strategic plan does not have relevant indicators or well defined targets; this makes it impossible to provide satisfactory accountability reports on performance with respect to the quality of childcare services. Value-for-Money Audit Fall

28 Chapter 5 Recommendations The Auditor General has made recommendations to the MFA. They are all shown to the right. The audited entity had the opportunity to comment on these recommendations; its comments can be found in paragraph We want to point out that it accepted all of the recommendations. Recommendations to the Department 1 In its process for allocating reduced-contribution spaces: Specify and adequately quantify the childcare network s needs and priorities, make them public and assess to what degree the developed spaces meet them; Disclose the eligibility and assessment criteria for applicants and projects; Use an assessment grid that makes it possible to classify projects on the basis of the selected criteria, to verify the information provided and to justify the acceptance or refusal of projects. 2 Implement and assess, according to a specific time frame, the measures needed to facilitate continuous quality improvement for all types of childcare services. 3 Ensure, in a timely manner, that the legal and regulatory requirements for quality are met, particularly the ratio of qualified staff members and the implementation of the educational program. 4 Obtain accountability reports from the coordinating offices on the quality of home childcare services and analyze the data to have a complete picture of the network and to ensure that the coordinating offices meet the requirements to which they are subject. 5 Plan and carry out inspections to ensure that uniform, economic, efficient and effective services are provided, particularly by: Determining the frequency of inspections as a function of risks; Providing work tools in a timely manner; Carrying out work in the time prescribed; Developing a follow-up system for non-compliance notices; Implementing measures for quality control and quality assurance of files. 6 Improve the non-compliance data that are published for each childcare service so that the available information is more useful to parents. 7 Improve its strategic plan by the addition of objectives and results indicators for the quality of educational childcare services. 8 Assess the level of success in meeting program objectives for educational childcare services, particularly with respect to the quality of childcare services, and make reports on the results. 22 Report of the Auditor General of Québec to the National Assembly for

29 Highlights Société de l assurance automobile du Québec : Compensation of Accident Victims 6 Audit Objective The purpose of our audit was to ensure that the Société de l assurance automobile du Québec (SAAQ) efficiently, economically and fairly manages files whose compensation costs may be the most signifi - cant in accordance with the Automobile Insurance Act and the applicable regulations. The SAAQ s responsibilities include compensating accident victims and facilitating their rehabilitation. In 2010, 100,981 claims were processed, and the amount paid to compensate accident victims were more than $990 million. In 2007, the SAAQ developed and implemented a compensation action plan to optimize its processes. Positive operational changes have been made. The Report is available at Audit Results We present here the principal findings of our audit concerning the compensation of accident victims in files judged to have a high risk for prolonged disability, files involving victims who were seriously injured or who died and files being part of the long-term category. No assurance that the information provided is always accurate. In 43 out of 85 files examined that involved people who were employed or unemployed but able to work at the time of the accident, there was no contact with the employer or other documentation except for the employer salary certification, to ensure that the information provided was accurate. The same was true for unemployed people who obtained a confirmation of hiring from the employer after the accident. To reduce the risk of incorrect information provided on forms, the SAAQ should identify situations in which risk may be higher and specify what documentation is needed in these cases. Files with a high risk of prolonged disability: the management framework is generally not respected and follow-up should be improved. In 77% of the 44 files examined, there was at least one key action indicative of deficiencies in the SAAQ action plan. As for follow-ups on interventions, the SAAQ has not set limits for treatments in private clinics beyond which control measures are required. For 14 of the 44 files examined, people had received 50 or more physiotherapy treatments. Among those files, there were 7 in which the number of treatments varied between 98 and 300 over a period of 12 to 14 months. Rehabilitation of seriously injured victims: little follow-up on the terms and conditions provided for in the administrative agreement with the Ministère de la Santé et des Services sociaux. For 16 files examined, several of the elements provided for in the agreement were not respected, including services invoiced but not provided for in the coverage plan and therefore not approved by the SAAQ. Computation of the residual pension: actual earned income is not always taken into account. Generally, the Société reviews an accident victim s compensation in the third year by carrying out an employment determination, whether or not the person is employed, and may then pay him or her a residual pension. A person holding employment in a category other than the one determined by the SAAQ may have an unfair advantage because his or her residual pension will not be reduced. No formal quality assurance process. The deficiencies noted in the report lead us to conclude that the SAAQ must increase its quality assurance efforts. That would certainly ensure that cases would be handled more uniformly and more fairly. Value-for-Money Audit Fall

30 Chapter 6 Recommendations The Auditor General has made recommendations to the Société. They are shown to the right. The audited entity had the opportunity to comment on these recommendations; its comments can be found in paragraph We want to point out that it accepted all of the recommendations. Recommendations to the Société 1 Review its methods to ensure that the information provided for computing compensation is accurate in situations judged to be at-risk. 2 Specify and implement the processes needed to ensure fairer treatment with regards to advances for permanent injuries and payment for temporary injuries regarding compensation for nonpecuniary damage. 3 Ensure that complete information is included in the action plan for high-risk compensation cases, that key actions are carried out in the allotted time and that an adequate follow-up on interventions and action plans is performed. 4 Analyze the relevance of establishing guidelines for certain medical treatments, to ensure a close follow-up on treatment plans. 5 Ensure that the terms and conditions provided for in the administrative agreement with the Ministère de la Santé et des Services sociaux on services offered in rehabilitation facilities are respected. 6 Analyze the fairness of the computation of the residual pension where an accident victim holds or does not hold the employment determined. 7 Implement adequate mechanisms to ensure the proper computation of residual pensions. 8 Implement adequate mechanisms to ensure treatment consistency where an accident victim has more than one active file. 9 Establish a formal quality assurance process to ensure the quality and the uniformity for the claim processing of the accident victims. 10 Adopt relevant indicators and relevant targets for all critical aspects of its performance. 11 Continue its efforts to have detailed and continuous information in order to monitor the factors that impact compensation costs. 24 Report of the Auditor General of Québec to the National Assembly for

31 Highlights Aide financière aux études 7 Introduction After an interval of a few years, we carry out a follow-up on our audit work. Our objective is to see if the recommendations covered by our follow-up have been taken into account and if the entities concerned have remedied the underlying deficiencies. The exercise completes the initial audit and makes it possible to inform the parliamentarians on the actions that have been taken to address the weaknesses that we reported. When problems remain, we briefly recall their causes and consequences, and we invite the entities to continue or intensify their efforts. The Auditor General s responsibility is to provide a conclusion on the objective mentioned above. To do so, we have collected sufficient and adequate audit evidence to reasonably support our conclusion and to obtain a high level of assurance. Our conclusion is based on the same evaluation criteria as those used for the value-for-money audit. 7.4 Follow-up engagements are carried out under the Auditor General Act and in accordance with the working methods in effect. Those methods comply with the standards for assurance engagements issued by the Canadian Institute of Chartered Accountants In this chapter, we give the results of the follow-up that we carried out with regard to student financial assistance. We completed our work at the end of June The initial audit was carried out from April 2006 to June 2007 with the Ministère de l Éducation, du Loisir et du Sport (MELS). Our objective at that time was to assure ourselves that the MELS manages the student financial assistance plan economically and efficiently and that it reports on its performance. The results of the audit were published in Chapter 2 of Volume I of the Report of the Auditor General of Québec to the National Assembly for At that time we made 16 recommendations to the Department. Our follow-up work focused on 10 of them. We chose those that, in our opinion, contribute the most to improve the management of the student financial assistance plan. We excluded the three recommendations for which the MELS needed to continue its efforts, because our initial report had already pointed out certain measures that the MELS was in the process of implementing. We also set aside the recommendation on reporting and those that concerned management information. Value-for-Money Audit Fall

32 Chapter To carry out our follow-up work, we met with managers and employees of the Department, and we examined various data and pertinent documents. Furthermore, for various purposes, we analyzed more than 50 files for which assistance was granted, mainly during the and award years. We also examined quality control activities carried out by the MELS. They concerned files that contained claims made by financial institutions and tasks carried out by the employees of the Department who are assigned to the collection of loans and bursaries. As a reminder, Aide financière aux études (AFE) is an independent service unit under the control of the MELS, and its mission is to facilitate access to studies by offering a financial assistance plan adapted to the needs of the student population. In this regard, it mainly administers the Loans and Bursaries Program, either for full-time secondary school vocational training or postsecondary studies. The AFE s main activities with regard to this program are awarding financial assistance and managing loans. Table 1 shows the financial assistance awarded by the AFE under the Loans and Bursaries Program during the award year, 1 which ended during our initial mandate, and during the award year, which ended shortly before we began our follow-up work. Table 1 Financial assistance awarded under the Loans and Bursaries Program (in millions of dollars) Loans Bursaries Total Number of applications 156, ,947 Number of beneficiaries 132, ,997 Source: MELS. 1. An award year extends from September 1st to August 31st of the following year. 26 Report of the Auditor General of Québec to the National Assembly for

33 Highlights General Conclusions 7.11 Our work leads us to conclude that there have been significant improvements in the management of the student financial assistance plan. In fact, 7 of the 10 recommendations have shown satisfactory progress (Table 2). However, the MELS will have to deploy additional efforts to improve the situation with respect to the following points: confirmation of academic information by educational institutions; assistance overpayments; information on student loans that financial institutions must provide to credit bureaus where the authorization in that regard has been signed. Table 2 Follow-up on the Recommandations Satisfactory Progress Unsatisfactory Progress Allocation of assistance Processing aid for the purchase of computer equipment Put in place controls to make sure that the loans allocated for the purchase of computer equip ment are used for this purpose. Educational institutions Make sure that educational institutions adequately fulfill their responsibilities when it comes to the verification and confirmation of the required educational information relating to students. Financial assistance paid Take the necessary measures in order to reduce the costs associated with assistance over-payments. Management of loans Risk analysis Make a risk analysis of the management of loans. Management of claims Specify to financial institutions the collection work that it expects from them and ensure the quality of such work. Specify to financial insti tutions the deadlines for submitting claims. Review the practice that consists of paying interest to financial institutions when they already are in possession of the sums. Collection of debts owed to the Department Manage more effectively the prescription of student debts. Make sure that the guidelines concerning the timing of interventions are respected. Make sure that financial institutions inform credit bureaus about student loans when the appropriate authorization form was signed. Total of Recommandations 7 3 Percentage of Recommandations 70% 30% Value-for-Money Audit Fall

34

35 Significance of the logo An easy-to-recognize building, Parliament, where the National Assembly sits. It is this authority which has entrusted the Auditor General with his mission and to which he reports. Three dynamic lines, illustrating: the three types of audits carried out by his staff, namely financial audits, audits of compliance with statutes, regulations, policies and directives, as well as value-formoney audits; the three elements that are examined during value-for-money work: economy, efficiency and effectiveness; the three fields social, economic and environmental related to the stakes concerning sustainable development. A truly distinctive sign, the logo of the Auditor General clearly illustrates that this institution, which is in constant evolution, aims to assist elected members in their desire to ensure the sound management of public funds, for the benefit of the citizens of Québec.

36 Auditor General of Québec The Auditor General of Québec reports exclusively to the National Assembly. This Assembly appoints the holder of this office upon a motion made by the Prime Minister and passed by at least two-thirds of the Members of the National Assembly. The term of office of the Auditor General is ten years and is not renewable. The Auditor General Act specifies the powers and duties associated with this office. The Auditor General fosters, through audit, the exercise of parliamentary control over the actions of the government, its departments and its agencies. He conducts financial audits, audits to ensure compliance of operations with acts, regulations, policies and directives, as well as value-for-money audits. The Auditor General carries out his responsibilities with respect to public bodies as well as government enterprises and agencies. He also has jurisdiction to audit the use of funds paid in the form of grants. Except for the obligations stipulated in the Act, the Auditor General determines the work that he performs and decides on the content of his report to the National Assembly. Mr. Renaud Lachance has held the position of Auditor General of Québec since August 9, He has a Bachelor s degree in Business Administration, a Master s degree in Taxation and a Master s degree in Economics. He is a member of the Ordre des comptables agréés du Québec. In 2010, the Ordre awarded him its most prestigious title, namely that of Fellow. From 1985 to 2004, Mr. Lachance served as a professor at HEC Montréal. He also assumed various leadership responsibilities in this university institution specializing in management.