RCUK Assurance. Quality Assurance and Validation of TRAC. March 2009

Transcription

1 RCUK Assurance Quality Assurance and Validation of TRAC March 2009 Page 1 of 75

2 Contents 1. Executive Summary... 3 Background...3 Overall findings...3 Key findings...4 Common areas of non-compliance...5 Good practice identified...5 Benefits of the QAV process...6 Future considerations...6 Acknowledgements...7 Conclusion: Introduction Aims and overview of the QAV self-assessment process Overview of the QAV process...11 Operation of the QAV process...12 Sample for QAV visits...13 Assessing the QAV process...14 QAV self-assessment returns...17 Benchmarking Common findings and good practice Impact of errors on the charge-out rates for research...21 Relationship between volume of RC funding and level of compliance...22 Summary of issues and good practice identified...23 Benefits of the QAV process Looking forward Tracking the QAV process...32 Existing developments...32 Development of TRAC...33 Provision of ongoing support for the sector...34 Suggested updates and amendments to TRAC guidance...34 Future assurance arrangements...39 Appendices Appendix 1 The QAV Assessment process...41 Appendix 2:- Institutions included in the QAV Process...43 Appendix 3:- QAV - Audit Programme...44 Page 2 of 75

3 1. Executive Summary Background The Transparent Approach to Costing methodology (TRAC) was introduced by UK universities following the 1998 Spending Review 1. In 2004 the Government announced that all research would be funded on a full economic cost basis, as from April Since that time, the level of funding provided to institutions by the Research Councils has been informed in part by institutional calculations of the indirect and estates cost of research activity, prepared under TRAC. The Quality Assurance and Validation of TRAC process (QAV) was designed to provide assurance principally to the Research Councils and by extension the Higher Education Funding Councils that the annual TRAC methodology for Research is being applied appropriately, and that all rates used in the funding of research activity are being calculated correctly. This report summarises the findings of the RCUK QAV review, including the visits to 50 institutions. The review was comprehensive and thorough. It provided a valuable insight into the current level of compliance with TRAC requirements across the UK higher education sector, allowing instances of good practice and common areas of non-compliance to be identified. The QAV process has been highly consultative and transparent with an emphasis on helping institutions further refine their systems and processes. Generous timescales were offered to allow the self-assessment process to be undertaken. RCUK Assurance undertook a thorough and comprehensive communication strategy highlighting the importance of the exercise. This was supported by letters from the Chair of the TRAC Strategy Group to Heads of Higher Education Institutions emphasising the importance of the robustness of data reported from TRAC. Many stakeholders and other partners have worked very closely together to enable the process to work successfully. This is due in no small measure to the support of HEFCE, KPMG, JM Consulting and the university sector at large. Overall findings The overall findings indicate that institutions continue to face substantial challenges in implementing TRAC methodology. TRAC and fec have been in place for over 10 and 4 years respectively, and whilst almost all the institutions visited were demonstrably committed to their continuing implementation, a third of the sample visited were found to have material instances of non-compliance with TRAC requirements. The report highlights a significant volume of non-compliance with TRAC. Although the material impact on the rates charges to the Research Councils is broadly neutral, the significant level of issues raised at institutions gives cause for concern and validates the need for the QAV process and a level of assurance in the future on high risk areas which have emerged. 1 See Page 3 of 75

4 There are however positive developments to recognise as we look forward: i) The QAV process does not conclude with the production of this report. All institutions which reported non-compliance are required to submit quarterly action plans to RCUK Assurance outlining the remedial action required to rectify these areas. ii) iii) iv) The quality of the annual TRAC returns to HEFCE in 2009 has been significantly higher than the previous year, when a significant proportion of institutions had to be contacted to review specific fields or systemic errors. The newly formed TRAC Development Group has successfully begun to address some of the strategic failings which have undoubtedly contributed in part at least, to the current position (E.g., TRAC guidance; review of Academic Staff Time). This group is already beginning to demonstrate dividends based on suitable representation from across the sector and stakeholders. Proposals in this report provide for the high risk areas identified to be incorporated into a new single assurance framework for the Research Councils within their existing Funding Assurance programme. This will also pursue the principle developed within QAV to encourage other funders to place reliance on the outputs of any future assurance work in this area. Key findings Average of 10 recommendations per institution Over 50% of institutions failing to meet requirements of TRAC across 7 main areas. On average, 10 recommendations were made at each institution visited. This was not unexpected given the complexity of the guidance. At the level of individual institutions the process identified a number of failures in the application of the TRAC methodology. There was a close correlation between institutions extent of compliance with TRAC requirements and volume of Research Council funding, with more findings reported at the less researchintensive institutions visited. The correction of identified issues has resulted in adjustments of up to 20% to institutions indirect cost rates, and up to 100% in the case of estates rates. A number of other issues were identified whose impact could not be quantified in the course of the programme, in particular those relating to academic time allocation and estates costs. Increased level of non-compliance at less researchintensive institutions Three institutions were placed into sanctions, suffering a 25% reduction in Research Council funding on the non-compliant rate until resolution of the identified issues could be confirmed. It should be noted that sanctions were only ever applied as a last resort in the QAV process with the benefit of the doubt always afforded to the institution. Page 4 of 75

5 Common areas of non-compliance The visits also highlighted common areas of non-compliance where further work is required by institutions to ensure their research charge-out rates are materially accurate. The main areas of non-compliance identified are summarised below, and details of the specific issues noted in relation to each area can be found in section 4. Robustness of academic staff time allocation data there was scope to improve the process of collecting and analysing academic time allocation data at 36 of the institutions visited (72%). Estates costs further work is required to develop the allocation of estates costs at 29 institutions (58%). Such work includes consideration of allocating the costs of circulation space and reducing the use of time allocation data or income to allocate the costs of shared space. Cost adjustments further refinement and correction of the calculation of the infrastructure and return for financing and investment adjustments was needed at 30 (60%) and 13 (26%) institutions respectively. Reasonableness checks and benchmarking while most institutions had undertaken some reasonableness checks on their data, there was scope to improve these at 39 institutions (78%) and in some cases significant errors on the returns were not detected in advance of the return being submitted. Attribution of indirect costs there remains room for further improvement in the choice and application of indirect cost drivers at 28 institutions (56%). In several cases the cost driver model had not been reviewed in a number of years. Calculation of estates charge-out rates there were opportunities to improve the process of identifying the staff FTE s for inclusion in the laboratory and non-laboratory estates rates at 28 institutions (56%). Major research facilities several instances were identified where institutions had chosen not to charge the full economic costs of facilities in costing research projects, as these were felt to be unacceptably high. Good practice identified While the QAV process has highlighted some notable issues with the application of the TRAC methodology, it should be stressed that it also identified areas of good practice, including: TRAC/fEC steering groups all but seven institutions have steering groups which meet regularly to oversee the development of TRAC and fec. Use of TRAC in resource allocation a minority of institutions are now utilising the TRAC methodology to inform their internal resource allocation process. Laboratory technicians despite this being the first year of its implementation, the vast majority of institutions had produced robust costings for their laboratory technicians. Page 5 of 75

6 Major research facilities the research-intensive institutions, in particular, have invested significant effort in developing detailed costings for both major and small research facilities. Whilst there will inevitably be future refinements, good progress has been made by many institutions. Benefits of the QAV process In addition to meeting its primary objective of providing assurance to the Research Councils and the Higher Education Funding Councils, the QAV process has delivered tangible benefits in the following areas: Providing a roadmap for institutional improvement; Promotion of continued accountability; Producing a tool that facilitated and empowered institutions to undertake their own selfassessment irrespective of whether they were subjected to a QAV visit. Adding value to the guidance by clearly describing the minimum requirements in TRAC for the first time; Introducing benchmarking information that is now embedded in institutional information systems thereby assisting with their own reasonableness testing and assurance; Reducing the audit burden for institutions applying the TRAC EC Framework 7 guidance in costing Framework 7 bids; Dissemination of good practice; and Identification of opportunities to enhance TRAC guidance. In addition to the benefits above, RCUK will run a number of roadshows in partnership with KPMG to publicise the experiences from the QAV visits more widely with the sector. Further detail on these benefits can be found in Section 4. Future considerations The QAV process has brought to light a number of issues which would merit further consideration by the Research Councils and the Funding Councils. These are detailed in section 5, and include: The need for the provision of ongoing support and sharing of good practice within the sector. A number of areas where review or clarification of the existing TRAC guidance would be of value to HEIs. Future assurance arrangements in respect of TRAC and fec. Page 6 of 75

7 Acknowledgements RCUK Assurance would like to record their sincere appreciation to KPMG who undertook the QAV audits with great professionalism and whose work has contributed significantly to the production of this report. RCUK Assurance would also like to record their sincere appreciation to all the universities and to TRAC staff in particular for their co-operation with the QAV programme, to the 50 institutions visited, to the Funding Councils and to JM Consulting for their help and assistance without which the successful completion of the process would have not been possible. Conclusion:- The QAV process has operated successfully and has been well received by the institutions that have received a visit. The process has been very intensive and thorough. It was nevertheless conducted in a pragmatic and constructive manner which has focussed on assisting institutions with the development of their processes to promote compliance with the TRAC requirements. The QAV process has provided a benefit to institutions and has made a number of suggestions for further improving the TRAC guidance, the support mechanisms for institutions and the future assurance arrangements. The newly formed TRAC Development Group is already making encouraging progress in reviewing these issues and will provide a good forum for taking forward the findings of this report. A large number of findings have been recorded and although only a small number of institutions have suffered sanctions as a result of a QAV visit, there were a significant number of material issues identified which, although largely cost neutral give cause for concern on the overall implementation of TRAC, particularly but not exclusively at the less research intensive universities. The limited assurance gained from the QAV process confirms the need for an ongoing external assurance process. Although there is no need to repeat the current QAV exercise, the optimum approach is to merge the assurance requirements of the high risks identified through QAV into the current Research Councils Funding Assurance Programme process as part of a single assurance framework. Page 7 of 75

8 2. Introduction (Background to the Quality Assurance and Validation of TRAC) 2.1 The Research Councils jointly provide 3bn of funding annually for Research and Postgraduate training in the UK. This funding is passed to a range of organisations and a significant proportion of the funding is awarded to Higher Education Institutions (HEIs). 2.2 For a number of years there has been a widely held view that the funding from a range of public funders did not cover the full cost of HEIs actually undertaking the research something that the Transparent Approach to Costing (TRAC) returns confirmed from the late 1990s onwards. 2.3 In view of this, the Government made additional funding available to the Research Councils, the Royal Society, the Royal Academy of Engineering, and the British Academy to meet 80% of the HEIs Full Economic Cost (fec). By this is expected to reach 516M per annum, with the level of funds growing to match the proportion of grants that are funded on a fec basis. 2.4 The Research Councils required cost based research on a fec basis at project level from 1st September 2005, with the first grants starting in April This included direct funding of academic resource for the first time. Since then, institutions own calculations of the fec of their research have directly informed the level of Research Council funding they receive. 2.5 A quality assurance process was implemented in 2004/05 on behalf of HEFCE. This consisted of a single day visit to each institution to offer a high level reasonableness review of the implementation of TRAC. It examined whether the correct methodology had been applied in deriving the estates and indirect cost values but did not examine the numbers or values produced out of these calculations. 2.6 In 2006, the then Office of Science and Innovation commissioned a report on how quality assurance and validation arrangements should evolve in the light of the move by the Research Councils to fund research on the basis of 80% of full economic costing. The report produced by Robson Rhodes recommended that the Research Councils should institute enhancements to their existing assurance programme to ensure that full compliance with the implementation of fec arrangements has taken place. 2.7 The proposals suggested that the existing Research Council assurance programme should be temporarily enhanced by a regime of external inspection. This regime would be directed towards providing assurance at an institutional level as opposed to project based that compliance with fec had been achieved. It would also be focused on supporting institutions in their adoption of fec, and enabling the sharing of best practice. 2.8 The seven Research Councils 2 created a new cross-council unit to develop an assurance strategy on their behalf. RCUK Assurance was therefore set-up to take forward a process for the Quality Assurance and Validation of TRAC (QAV). 2 The seven Research Councils are: Arts and Humanities Research Council, Biotechnology and Biological Sciences Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Medical Research Council, Natural Environment Research Council, Science and Technology Facilities Council Page 8 of 75

9 2.9 The primary objective of QAV was therefore to provide assurance to the Research Councils that the annual TRAC methodology was being applied appropriately and that all rates used in the funding of research activity were being calculated correctly In 2007/08, the Research Councils (RCs) paid 1.3bn in respect of project based research grants and fellowships. At the present time, there remains a mixed portfolio of pre-fec and post-fec grants active at research organisations. By the end of 2009/10, it is likely that nearly all active grants will have been awarded on a fec basis. The chart below demonstrates the materiality of the awards by expenditure type and thereby the relative importance of estates costs and indirect costs calculated in accordance with TRAC. Within the next two years, the costs charged for indirect and estates rates are likely to account for 40% of all cash paid by the RCs annually, a figure in excess of 0.5bn. Figure 1:- Proportion of funding paid in 2007/08 by fund category Exceptions, 79M 6% Indirect, 377M 30% Directly Incurred, 582M 47% DA Other, 103M 8% DA Estates, 114M 9% 2.11 The QAV process was therefore key to providing the required assurance over these and other new fund categories. QAV has been undertaken to provide assurance to both the RCs and the Funding Councils regarding the charge out rates for research and for the data included in the annual TRAC return RCUK Assurance undertook a comprehensive consultation process with the sector to determine the optimum approach for this exercise. This included the following representative groups: British Universities Finance Directors Group (Costing and Pricing Sub-Group) Funding Councils Council for Higher Education Internal Auditors (CHEIA) TRAC regional groups National Audit Office Association of the Heads of University Administration (AHUA) Higher Education Regulation Review Group (HERRG) The Brunswick Group, the Association for Research Managers and Administrators & other institutional groups Page 9 of 75

10 2.13 Three mini-pilot visits and subsequently three full pilot audits were undertaken at research intensive institutions. These helped formulate a process which offered assurance to the Research Councils and which offered institutions a self-assessment tool with which they could establish their own level of compliance. This would be particularly helpful for those universities which would not later be subject to full QAV audits Following the QAV pilots, a series of roadshows were held by RCUK Assurance and JM Consulting to launch the QAV process and ensure effective engagement with institutions. To further assist the sector in responding to the QAV approach the QAV review audit programme was made available on RCUK Assurance web site. It is attached to this report as Annex KPMG were appointed through a competitive tendering process to undertake the QAV audits on behalf of the Research Councils An important driver was to minimise any duplication of audit on institutions. Consequently, RCUK Assurance secured the agreement of HEFCE to amend the 2006/07 annual TRAC return to include greater detail, specifically regarding the indirect costs of research. For the purposes of this process, the modified document was referred to as QAV-DATA. The 2007/08 TRAC return requests much of the same information again as QAV-DATA to support RC benchmarking analysis. The timing of the Research Councils information needs was aligned to that of HEFCE to further minimise reporting requirements and burden on institutions QAV is underpinned by two other documents: QAV-PROCESS, which summarises the requirements, protocol and timeline of the course of action in detail for RO management and QAV-METHOD which summarises the minimum requirements of TRAC in one short summary document; the first time such a summary had been produced. The latter document has been welcomed by users of TRAC as a helpful new initiative Research organisations from the HE sector that do not receive publicly funded research on a five year average in excess of 0.5m are not required to calculate their own rates within TRAC. Sector average rates referred to as compliant default or dispensation rates can be applied to any proposals completed for RC funding. Non HE Institutions that are eligible to apply for RC funds but cannot calculate their own rates also use the compliant default or dispensation rates. Neither of these groups of organisation were required to participate in the QAV process. Page 10 of 75

11 3. Aims and overview of the QAV self-assessment process 3.1 Following the consultation process, RCUK Assurance developed a self-assessment mechanism for institutions with the following aims and objectives: To provide a mechanism for institutions to ensure that the annual TRAC methodology is being applied appropriately, and that all rates are being calculated correctly; To provide assurance to the Research Councils on the application of TRAC and the calculation of the cost rates; To provide assurance to the Higher Education Funding Councils that they can place reliance on the figures in the annual TRAC return; and To be an accepted basis for the audit of certain aspects of the TRAC EC Framework 7 requirements. 3.2 To complement the QAV process, the Research Councils also operate the Funding Assurance Programme (FAP). Further detail on both processes can be found at Overview of the QAV process 3.3 The diagram below provides a broad overview of the QAV process, from the initial submission of an institution s cost rates and self-assessment (in the form of the QAV-DATA and QAV-METHOD documents), to completion of the QAV review, including potential follow up visits. 3.4 As this overview makes clear, the process required input from a range of interested parties, including all the Funding Councils, the Research Councils and JM Consulting and institutions themselves through the pilot visits that were undertaken. An open and transparent approach was therefore critical from the outset, and all parties deserve credit for the way in which this was achieved. Overview of the QAV process Institution QAV review team QAV review Possible follow up HEI Self assessment and data return Submitted through HEFCE Review by HEFCE & RCUK Benchmark feedback No further action RCUK Technical support JM Consulting Page 11 of 75

12 3.5 The successful completion of each stage of the QAV process was also dependent on a range of other factors, including: Timely submission of data by institutions; Effective analysis of the data and assessment of risk by RCUK Assurance and HEFCE; Institutional co-operation in the planning and scheduling of reviews; Flexibility in the staffing of reviews on the part of KPMG; A high level of commitment and assistance from institutions during the visits themselves; Robust quality control by KPMG to ensure consistency of approach across all the institutions visited; and Effective liaison between KPMG, RCUK, JM Consulting and institutions in the resolution and reporting of issues. 3.6 These criteria were met in full. This allowed the QAV process to meet its overall objective of delivering assurance on institutions compliance with TRAC and the robustness of their charge-out rates for research. Operation of the QAV process 3.7 The methodology that underpins TRAC is complex and requires a series of informed judgements by universities as such the risk of error and non-compliance is increased. At the same time, however, the Research Councils recognised the need for a proportionate and risk-based approach which would minimise disruption to the sector. 3.8 To enable such a process to be developed, the QAV review was piloted at the Universities of Liverpool, Manchester and University College London in advance of its formal launch in September This was an invaluable process as it enabled the approach to be tailored and adapted such that it was suitable and acceptable to institutions. We are very grateful to the three institutions involved. 3.9 The QAV process therefore entailed submission of the TRAC data and cost rates in a prescribed format (the QAV-DATA form) and completion of a brief self-assessment exercise by all universities (QAV-METHOD Part A). In addition the institutions in the top 60 by volume of Research Council funding were required to complete a more detailed selfassessment (QAV-METHOD Part B) Institutions assessed and graded their performance based on materiality of variances against the TRAC requirements. A three rating would be recorded if an error affects any rate by more than 10% and a two rating if the effect on these rates was greater than 5%. Further details of the process and the report gradings used are in Appendix This was followed by visits to 50 institutions in England, Wales, Scotland and Northern Ireland by KPMG. In each case, the application of the TRAC methodology and the calculation of the cost rates by the institution were reviewed in detail. The visits, restricted to three days in length, involved meetings with key members of staff, including senior management, and the testing and verification of each institution s TRAC model and project costing methodology. Page 12 of 75

13 3.12 On completion of the visit, KPMG s findings were graded on a three point scale shown in Appendix 1, according to their estimated financial impact on the institution s cost rates. Draft reports were issued to institutions to allow them to confirm factual accuracy, quantify the financial impact of issues, and provide management responses to the recommendations made. Thereafter, a final report was issued to each institution by RCUK Where significant issues were identified, provision was made within the process for follow up visits to be undertaken by KPMG. Consequently, visits to eleven institutions are in the process of being undertaken The QAV process also placed a requirement on University Executive Committees to actively track progress against the implementation plan, and to submit an updated selfassessment to RCUK every three months where issues were identified at levels 2 or 3. This provides a mechanism for 35 of the institutions visited to move towards full compliance over time and importantly tracks issues through to their successful resolution. Sample for QAV visits 3.15 The visit sample was selected based on a number of relevant metrics including the materiality of Research Council funds received by an institution in 2006/07; a review of the rates calculated for QAV-DATA; an analysis of rates year on year; and the QAV-METHOD returns. Consideration was also made for those institutions that did not return either document in a timely manner Fifty institutional audits were undertaken, comprising 33 from the top 60 research intensives and 15 from the remaining pool of universities. A further 2 were identified later in the process, making 50 visits in total By the time the sample had been selected and the QAV audit element of the review was started, the RCs were able to identify the amount of funding ( ) paid in 2007/08 financial year (April March). The coverage of the sample obtained is recorded in fig. 2. Figure 2:- Coverage obtained by QAV audit sample based on 07/08 data ( m) Total Funding 1,243 Tested Institutions 1,030 Coverage (%) 83 Untested Balance 213 Page 13 of 75

14 Figure 3:- Coverage obtained by QAV audit sample by Research Council ( m) Research Council Total funding in 2007/08 Sample incl. pilots Percentage of sample (%) AHRC BBSRC EPSRC ESRC MRC NERC STFC TOTAL 1,243 1, AHRC, ESRC and NERC show a marginally lower percentage coverage when compared to other RCs. The ESRC and NERC fund a greater proportion of non-he Institutions which are not included in this sample at 4% and 7% of their budgets respectively. For AHRC, three institutions in their top 20 (being between 10 th and 20 th ) were not selected in the sample; these represent 6.5% of the total funding in 2007/ The sample also provides adequate coverage of the spread of ratings that were completed from the population of self-assessment. The sample gave a good representation of the spread of ratings in the population with a slight bias towards two and three ratings For the first year of detailed review, the sample provides a high percentage of coverage against the total Research Council funding that has been distributed. The sample has included institutions which are both research intensive and non-research intensive It should be noted however that it was important to ensure that institutions recording all 1 ratings at self-assessment were scrutinised, as there was the potential for material error to pass unnoticed. One such institution was indeed subjected to sanctions following a QAV audit where their rates were found to be materially overstated. Assessing the QAV process 3.22 The steps involved in delivering the QAV reviews of institutions are shown on the next page Steps 3, 4 and 5 of the process involved extensive liaison with individual institutions in the planning, delivery and reporting of the 3-day visits. The following pages provide a brief commentary on the operation of this aspect of the process, and its impact on the institutions visited. Page 14 of 75

15 6. QAV Project Report 2. Data analysis and risk assessment 5. Quality review and reporting 1. Project Management 7. Communication 3. Planning and Scheduling 4. Undertaking the visits Stage of the QAV process Planning and scheduling Undertaking the visits Commentary In the vast majority of cases institutions were highly committed to the QAV process, and were proactive in working with the QAV review team. This was reflected in the willingness of senior managers to make themselves available for the kick-off and debrief meetings with the QAV review team. Institutions were keen for the QAV visits to take place as early as possible, and although not all such requests could be accommodated the vast majority of visits were scheduled in by late September, two months ahead of schedule. Most institutions, though by no means all, had reviewed the QAV benchmarking prior to the visit of the QAV review team. However, not all institutions could readily provide evidence to demonstrate the validity of their outlying results. Thorough preparation for the process proved invaluable, with the best-prepared institutions having reviewed each element of the QAV audit programme in advance, and prepared a file of responses and supporting evidence prior to the review team s arrival. Formal documentation of thought processes, and the regular review of judgemental decisions, also proved highly beneficial to institutions in responding to the review team s queries. Successful completion of the visits in the 3 days available was a frequent challenge for the review team, particularly in the case of research intensive institutions with complex or self-built costing models and large numbers of research Page 15 of 75

16 Stage of the QAV process Quality review and reporting Commentary facilities. The process also proved to be very intensive from the perspective of institutions. In many instances, TRAC managers within institutions worked out-of-hours to deliver the information required, and this level of commitment to the process was greatly appreciated. In 4 cases, a lack on preparedness on the part of institutions necessitated the abortion of a visit. In all other cases however the on-site visits were successfully completed after the initial visit. Where there had been a recent change in the role of TRAC managers this often created difficulties in the provision of the necessary evidence and explanations, underlining the need for institutions to maintain comprehensive procedures and documentation. The QAV process highlighted a large number of issues at most of the institutions visited, and the process of resolving and reporting these issues proved challenging and timeconsuming in a number of cases. Issues such as a failure to consider circulation space or use of inappropriate cost drivers often required significant additional work by institutions before their impact could be quantified. This led to delays in providing the draft reports to institutions. In the case of other issues, most notably in the areas of estates and academic time allocation, there was often no means of quantifying the impact of issues until the completion of a full cycle under a revised methodology. As a result of the above factors the process of grading issues and finalising reports frequently took some weeks or months, and institutions patience in this respect was greatly appreciated. Overall, the approach taken by RCUK and the QAV review team to the finalisation of reports and the potential application of sanctions was sympathetic and pragmatic, with a clear emphasis placed on proactively working with institutions to achieve full compliance, rather than a heavyhanded application of the rules. This was reflected in the fact that while potentially material instances of noncompliance were identified at around a third of the institutions visited, only three were formally placed in sanctions. Page 16 of 75

17 QAV self-assessment returns 3.24 Figure 4 below summarises returns from all universities by category where their level of non-compliance is greater than 5%. Where this applies to over 20% of all universities, this has been classed as high risk. Where it applies to 10%-20% of institutions, this is reported as moderate risk. Figure 4:- Summary of self-assessment questions identified as High Risk Ref. Question relates to- Total Summary risk rating Top 30 Summary risk rating A6 Estates costs High Low A9 Minimum requirements of TRAC High High B A1 Integrity of the Systems High Moderate B A2 Management High High B A3 Tests for reasonableness High High B B4 Academic staff time 1 Moderate High B C1 Full-time equivalents (staff numbers) High High B C3 Laboratory technicians charge-out rates High Low for Research B- C4 Research facilities High High B- D1 Academic staff costs (DA) High Moderate 1 This is the only question where the risk is greater in the top 30 compared to the relevant population 3.25 The top 30 institutions by research funding in general have self-assessed their performance as better than that of the total population. The only question which has showed the top 30 performing less effectively than the remainder of institutions was Part B question B4 for academic staff time. This finding is cause for some concern as this is a key driver within the TRAC process. The QAV visits identified the extent of non-compliance with this element of the TRAC guidance and the implications these have for calculating the charge out rates When looking at the total population, other questions show only low levels of noncompliance. However, the returns showed that 54% of institutions recorded all one ratings for part A and this figure dropped to 38% in conjunction with part B as well. Organisations recording ratings at levels two or three within their self-assessment account for just over half of the funding allocation in 2006/ As noted above, the output of self-assessment shows that the institutions have assessed themselves quite favourably; of the assessments received, only 12 institutions rated themselves with any level three ratings. Three of these research organisations were in the RC top 60. This level of error is not material for the Research Councils, and these Universities are expected to improve this position as part of the QAV process in a timely manner In many ways, these were encouraging results for the RCs, suggesting that TRAC was becoming embedded into the university accounting systems. The programme of visits was therefore essential for ensuring that these self-assessments were completed appropriately. The extent to which the self-assessment was indeed completed accurately was identified in Page 17 of 75

18 the QAV visits. Both processes together gave a clearer indication of the progress that institutions have made The elements of the QAV-METHOD showing a higher level of risk to the Councils will not all have an equal impact on the rates. For example, questions from part B section A (Integrity of the systems, management and test of reasonableness) are not likely to actually affect the TRAC rates; these tests require the institutions to review their own processes and complete benchmarking on their data to confirm its reasonableness. Low self-assessment here indicates a poor control environment but does not, in itself lead to errors in the TRAC output When reviewing the output from the Part A returns (minimum requirements of TRAC summary), it appears that smaller institutions struggle to meet with the requirements of the estates data calculations in TRAC. It is also worth noting the relatively high number of institutions of all sizes that record a two rating regarding adherence to the minimum requirements of TRAC From Part B (detailed self-assessment for research intensive universities), the poor returns regarding the calculation of FTEs and academic staff time are of most concern as noted in para In particular, academic staff time is an important cost allocation tool within the TRAC process. Indeed, the pilot QAV reviews undertaken also highlighted this as an area that many institutions may struggle to comply with TRAC for at present Overall, the self-assessment process has shown a high number of 1 ratings across the board Figure 5:- Distribution of self-assessment ratings Average Rating Part A (%) Part B A (%) Part B B (%) Part B C (%) Part B D (%) Benchmarking 3.33 Review of the QAV-DATA returns was undertaken by HEFCE as part of a data cleansing exercise; a number of institutions were contacted to review specific fields either for manual or systemic errors. Going forward, benchmarking analysis will provide an important element of the assurance process The benchmarking data is distributed to all institutions following analysis by HEFCE, to allow some introspective review to be undertaken after the submission of QAV-DATA. Institutions had the opportunity to re-submit their QAV-DATA during the year, if they identified any changes as a result of more detailed benchmarking data or otherwise Whilst benchmarking identified a number of outlying institutions, the overall trend across the sector for rates was relatively consistent. The main noticeable trend is the greater number of outliers amongst the less research intensive universities. Benchmarking has provided encouragement to the RCs regarding the consistency of data, however, masked within the chart below, are a significant number of institutions that required further contact to ensure that the calculations were undertaken correctly within the TRAC model. Page 18 of 75

19 Figure 6:- Chart shows the average charge out rate by peer group, error bars identify upper and lower quartiles (2006/07 TRAC data) 50,000 40,000 /FTE 30,000 20,000 Indirect rate Estates lab rate Estates non-lab rate 10,000 0 Group A Group B Group C Group D Group E Group F Group G UK Sector Peer Group 3.36 The RCs also noted during review of the QAV-DATA that there were a number of institutions recording a greater deficit against RC funded research than some other research funding streams including the EU, charities 3, other government departments and industry. Whilst there is a mixed economy of pre-and post-fec grants active currently, in many of these cases the RCs expect to be funding at least comparable with these other funders. 3 Seven out of the top 10 universities reported a greater deficit on RC funding than EU grants; 4 out of the top ten reported a greater deficit on RC funding than Charities Page 19 of 75

20 4. Common findings and good practice 4.1 The QAV visits assessed institutions compliance with TRAC requirements in the following principal areas: Academic staff time; Estates costs; Cost adjustments in respect of infrastructure and the return for financing & investment (RFI); Attribution of indirect costs; Full-time equivalents for staff and students; Calculation of the charge-out rates for research. 4.2 Where institutions were found not to be in compliance with these or other aspects of TRAC guidance, the severity of the issue was graded on a scale of 1 (minor), 2 (significant) or 3 (significant and material) which reflected the impact that the issue had on the cost rates (further details are outlined Appendix 1 of this report). 4.3 For the vast majority of institutions visited, the QAV process highlighted multiple instances of non-compliance with TRAC requirements, with a total of over 500 recommendations made in the course of the 50 visits. This translated to an average of 10 recommendations per institution, although the number of recommendations ranged from as few as 2 to as many as The majority of issues raised (72%) were graded as level 1, representing either issues that had only a minimal impact on the charge-out rates, or opportunities to adopt good practice. Only 9% of issues were classed as level 3 (significant and material) and 19% were classed as level 2, or significant. 4.5 Overall, just under half of the institutions visited, 23, were found to have level 3 issues, while two thirds, or 34, had level 2 issues. 4.6 The graph on the following page provides an overview of the most common issues identified from the QAV process, highlighting the number of issues identified in each case and their significance. 4.7 As the graph illustrates, the most common issues were found in the following areas: Time allocation survey; Infrastructure adjustment; Reasonableness checks and benchmarking; Calculation of research charge-out rates; and Systems and documentation. Page 20 of 75

21 Summary of Issues identified Num ber of issues Level 3 Issues Level 2 Issues Tim e allocation survey In fra stru ctu re adjustm ent Reasonableness checks and benchm arking Calculation of research charge-out rates System s and docum entation E state s costs Attribution of indirect costs M anagem ent and oversight M ajor Research F a c ilitie s RFI adjustm ent Student FTE's Charging Academ ic staff costs (DA) to projects Com pletion of QAV- DATA form D ire c t a ttrib u tio n o f costs In d exa tion rate s Level 1 Issues 4.8 Other issues with a potentially significant impact on charge-out rates included estates costs, the RFI adjustment, and the calculation of rates for major research facilities ( MRFs ), where small changes in approach often had an impact on the charge-out rates for individual facilities. Further detail on the specific findings noted in each of these areas can be found below. Impact of errors on the charge-out rates for research 4.9 As previously noted, the QAV visits identified a wide range of areas where further work is required to demonstrate full compliance with the TRAC requirements. In many cases further investigation revealed that these instances of non-compliance had not resulted in material errors, where material is defined as changing a research charge-out rate by more than 10%. In other cases, particularly the time allocation survey, it was generally not possible to quantify the impact of non-compliance. Page 21 of 75

22 4.10 This left five areas of non-compliance which were confirmed as having a material impact on the rates for at least one institution. The impact of these areas on the estates and indirect rates is illustrated in the graph below, both individually and in aggregate The graph shows the extreme impacts that these issues had in terms of the cost rates. It also shows the average increase / decrease in the cost rates as a result of each issue. Please note that there were a number of cases where institutions were unable to quantify the impact of issues such as estates allocations and TAS. As such, these are not included in the graph on the following page: 4.12 Overall, the impact of the quantified errors was to increase the estates laboratory rate and decrease the indirect rate, on average, with no significant change in the estates nonlaboratory rate. Relationship between volume of RC funding and level of compliance 4.13 The final notable finding of the QAV process was a correlation between the volume of Research Council funding received and the extent of compliance with TRAC requirements. Illustrated by the graph below, this is likely to reflect the greater priority and resources attached to the process in the more research-intensive institutions. In smaller institutions Page 22 of 75

23 where dedicated TRAC resource is more difficult to justify, achieving full compliance is correspondingly more difficult achieve. Average Number of Issues by Value of Research Council Funding Average Number of Issues Level 3 Issues Level 2 Issues Level 1 Issues 2 0 > 50m 30m - 50m 15m - 30m 5m - 15m < 5m Value of Research Council funding (2007/08) Summary of issues and good practice identified 4.14 The table below outlines the typical issues noted in relation to each area of weakness summarised by KPMG, and identifies relevant examples of good practice in each case. Area Common issues Good practice examples 1. Time allocation survey Concerns were commonly noted in relation to time allocation surveys, which tended to cast doubt on the validity of institutions TAS data, given the level of cost driven by the TAS. The impact of this was often difficult to quantify due to the need, in most cases, for the collection of a full year s data under a revised methodology. However, given the level of cost that is allocated by the TAS, there is reduced assurance that the charge out rates are correct in these cases. Specific examples of the issues found were as follows: 26% of institutions accepted TAS responses more than 2 months after the end of the period to which they relate. 14% of institutions had used time allocation data from previous years A number of institutions had devised innovative means of improving response rates to the time allocation survey. These included on-line systems which automatically generated reminders to staff who had failed to complete their returns, distribution of leaflets among staff to raise awareness, and the formal devolution of responsibility for chasing returns to School/Dept administrators. Page 23 of 75

24 Area Common issues Good practice examples without a process for reviewing its ongoing validity. 14% of institutions had undertaken TAS on a sample basis but failed to involve a statistician in the design and/or analysis of the survey, calling their results into question. This created uncertainty over the validity of their results. Limited validation and reasonableness checks on TAS data had been undertaken in 20% of cases. The weighting of the time allocation survey results for salaries/holidays/sickness/fte's had not been considered at 16% of institutions. Finally, 12% institutions had issues with low TAS response rates, or failure to adequately track response rates. 2. Estates costs One of the most significant issues in terms of impact on the cost rates was estates costs, with the following recurrent concerns noted: 42% of the institutions visited had not formally considered the impact that their basis of allocating circulation space had on the estates costs. 26% of institutions lacked robust information on the use of shared space, and were using time allocation data, income or other proxies to allocate the costs of this space. The approach to the weighting of space was found to be inadequate in 10% of institutions. In a small number of cases, institutions had extended the time allocation survey to nonacademic staff. Whilst not a requirement, this provided robust data for use in the allocation of support staff costs, and was felt to be helpful in raising the profile of TRAC among nonacademic staff. A number of institutions had a structured approach to engaging with the academic community to maintain the profile of TRAC and raise awareness of the positive impact that TRAC is having. Well-prepared institutions had undertaken work to consider the potential impact of alternative methods of allocating their circulation space prior to the QAV visit. This meant they were able to justify the choice of methods used to the QAV review team, and/or demonstrate that the impact of alternative approaches was not significant. 3. Infrastructure adjustment A number of institutions had made errors in their calculation of the infrastructure adjustment, as follows: In 38% of cases land, assets under construction and/or equipment had been incorrectly included in calculating the adjustment. Just under a third of institutions (30%) The majority of institutions were found to have used the templates provided in the guidance as a basis for the calculation of the adjustment. Page 24 of 75

25 Area Common issues Good practice examples 4. Reasonableness checks and benchmarking had failed to correctly adjust for depreciation on historic buildings. Insufficient evidence was retained to support a nil or unusually high longmaintenance adjustment at 20% of institutions. There was confusion over the correct basis for the insurance valuations used in the adjustment in 20% of cases, and instances of human error in preparing the adjustment were noted on 10% of the visits. While most institutions had undertaken some form of reasonableness checks, there was scope to improve these in 78% of cases. For example, it often appeared that TRAC figures were produced too late to allow for sufficient analysis to be undertaken, or for outlying results to be investigated and corrected. As a result, we noted several instances where returns were submitted containing known errors, which were only rectified via a resubmission some weeks or months later. In a number of cases, errors in the model or in the application of TRAC guidance could have been identified from a more detailed review against an institution s peers, or even its own prior year results, but went undetected. The ability to produce a draft TRAC return by late- December or early January is key, in that it then allows sufficient time for senior management to review, challenge and require amendments to the figures, if necessary. 5. Calculation of research charge out rates While 90% of institutions were able to provide a robust basis for their staff FTE figures, the use of these figures in the calculation of the cost rates often proved troublesome. Specifically: The impact of off-campus staff on the calculation of the estate rates had not been formally considered by 34% of institutions visited. 28% of institutions had not adequately considered the impact of desk-based work in laboratory departments, and vice-versa. The impact of this on the rates was often found to be significant Page 25 of 75