A Framework for evaluating the implementation of Private Finance Initiative projects: Volume 1

Transcription

1 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1 REPORT BY THE NATIONAL AUDIT OFFICE 15 May 2006

2 For further iformatio about the Natioal Audit Office please cotact: Natioal Audit Office Press Office Buckigham Palace Road Victoria Lodo SW1W 9SP Tel: Natioal Audit Office 2005

3 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1 This volume has bee published alogside a secod volume A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 2

4 This documet has bee prepared by the Natioal Audit Office as part of its Good Goverace programme. Good Goverace projects aim to draw o the NAO s cross-govermetal experiece ad idepedet perspective to help orgaisatios deliverig public services achieve excellece i fiacial maagemet, operatioal efficiecy ad quality of service. I particular, Good Goverace projects aim to: Promote improved maagemet of risks to the effective delivery of public services ad stewardship of resources; Focus o issues that, i the judgemet of the cliet orgaisatio, are timely, relevat ad importat; Cotribute to the delivery of efficiecy savigs, ad, where possible, direct fiacial impacts; ad Brig a wider perspective to the cliet orgaisatio s practice by drawig o the NAO s kowledge of fiacial maagemet ad service delivery issues across govermet. The Natioal Audit Office study team cosisted of: Sophia Colligwood, Louise Yaxley, David Jackso, Haah Paye, Susa Brow ad Christopher Hederso uder the directio of James Robertso. cotets preface 4 Part 1 A updated approach to assessig 5 whether PFI projects are beig implemeted well A updated approach to assessig PFI s value of 5 moey has become ecessary Our updated framework makes use of 6 top-level idicators The matrix framework is iteded as a tool to 7 assess PFI project implemetatio but is ot a replacemet for official guidace This report ca be foud o the Natioal Audit Office web site at

5 Part 2 Makig use of the matrix: 12 the audit criteria 2.1 Strategic Aalysis Tederig Cotract Completio Pre-Operatioal Implemetatio Early Operatioal Mature Operatioal 16 ANNEX Applyig the Matrix: Cosideratios for Evaluators 17

6 preface Preface This report sets out the issues that eed to be cosidered i evaluatig whether PFI projects have bee well thought through ad the implemeted effectively. Such cosideratios are cetral, although ot exclusive, i assessig whether good value for moey is beig obtaied from PFI projects. A umber of other issues icludig a compariso of what performace at what cost might have bee obtaied over a typical 25 to 30 year PFI cotract period from other procuremet routes are also key, although somewhat hard to assess. I this report, we preset a evaluatio framework coverig the lifecycle of projects from iitial strategic aalysis to the mature operatioal phase, i terms of idicators that spa six key busiess maagemet themes. It builds o ad updates our previous report i 1999, Examiig the value for moey of deals uder the Private Fiace Iitiative. Sice the, a substatially greater umber of deals have bee closed ad a growig proportio are ow i the operatioal phase. With the accumulatio of experiece it is possible to address the evaluatio of PFI projects i this phase. Part 1 describes the updated approach to assessig PFI project implemetatio. Part 2 outlies the evaluatio approach that ca be used i practice. Volume 2 provides full details with Aex 1 of that volume providig a guide to other govermet documets that address the desig ad implemetatio of PFI projects. The Framework is iteded for use primarily by evaluators, but it will also be of assistace to those implemetig PFI projects o a day to day basis. I additio, may of the issues cosidered are applicable to o-pfi procuremets. Evaluatio is ot a static activity however, ad readers eed to be aware of ew or revised guidace ad the results of detailed assessmets of PFI projects i carryig out this work. This Report is i lie with developmets up to ad icludig Budget A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

7 part oe Part oe A updated approach to assessig whether PFI projects are beig implemeted well This part of the report outlies a updated approach for assessig whether PFI projects are beig implemeted well, across all phases of the project life-cycle, accordig to a set of key busiess-maagemet imperatives. It also sets out how the framework ca be applied by project maagers, Private Fiace Uits ad exteral evaluators. A updated approach to assessig PFI s value of moey has become ecessary 1.1 I 1999, we reported o how we examied the value for moey of deals reached uder the Private Fiace Iitiative 1. The aalysis was based o four pillars, each of which comprised a umber of detailed aspects. The four pillars were: make the project objectives clear; apply the proper processes; select the best available deal; ad make sure the deal makes sese. 1.2 That framework s primary focus was o issues that arise durig the procuremet ad egotiatio of deals, rather tha o those arisig durig the life of a cotract. At the time of our 1999 report, few deals had reached a established operatioal phase ad it was ot the possible to exted the framework to cover this aspect. The framework outlied i this report builds o but replaces the four pillars. There is o existig framework or guidace to assess deals i operatio 1.3 The Treasury ad the Office of Govermet Commerce (OGC) have produced substatial guidace to assist Public Authorities i the procuremet ad evaluatio of PFI projects, icludig the 2004 Treasury publicatio Value for Moey Assessmet Guidace which set out a ew approach to assessig the value for moey of PFI 2. Most of the curret advice to Authorities also relates, however, to the procuremet of a PFI project ad does ot comprehesively address the operatio of the subsequet services, which are typically provided over a 30-year period. Achievig value for moey from PFI depeds as much if ot more o gettig the required operatioal performace as o gettig the best deal. Exteral iterest i how to assess the implemetatio of projects is high 1.4 Discussios with exteral parties i the public ad private sectors suggested to us that there was a substatial demad from those ivolved i desigig ad implemetig PFI deals for a overarchig framework for assessig whether PFI is beig implemeted effectively, particularly i relatio to a project s operatioal phase. The Treasury s 2003 publicatio Meetig the Ivestmet Challege also poited to the eed for such work 3. 1 Natioal Audit Office: Examiig the value for moey of deals uder the Private Fiace Iitiative (HC ). 2 HM Treasury: Value for Moey Assessmet Guidace (August 2004). 3 HM Treasury: Meetig the Ivestmet Challege (2003). A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1 5

8 part oe Our updated framework makes use of top-level idicators 1.5 I buildig the aalytical framework, we asked what processes ad outcomes are eeded to achieve a successful PFI project. We did this i part by reviewig the work i over 50 NAO reports o PFI ad PPP projects that we have published. Our aalysis the drew heavily o discussios with those i the private ad public sectors icludig with Private Fiace Uits i four govermet departmets, HM Treasury, Parterships UK ad 4ps. I additio to existig official guidace, we also drew o the Treasury Budget 2006 publicatio, PFI: Stregtheig Log-Term Parterships, which focuses o improvig value for moey especially i maagig the operatioal phase of PFI projects 4. The idicators are top-level statemets of what should be achieved to deliver ad ru effective PFI projects 1.6 Our framework comprises a series of maagemet themes which spa the various stages i the life of a PFI project, from Strategic Aalysis to Mature Operatio. The essetials for evaluatio for each maagemet theme at each life cycle stage are summarised by high level idicators oted at Figure 1, each of which is supported by hierarchical sub-idicators. The further dow the hierarchy, the more specific ad detailed the subidicators become. Project implemetatio ca be divided ito six phases 1.7 We idetified six distict life-cycle phases through which a PFI project passes, each ivolvig a specific ad separable set of issues that eed to be cosidered. These six idividual phases are: 1 Strategic Aalysis. This phase covers the steps followed by a procurig Authority up to outlie busiess case ad the decisio to procure through a PFI solutio. This phase aligs with the Stage 1 (the ivestmet decisio to go ahead with the project) ad Stage 2 (the decisio o how to procure the project) processes i the 2004 Treasury VFM guidace. 2 Tederig. This covers the steps udertake up to the selectio of a preferred bidder. This phase is equivalet to Stage 3 i the 2004 Treasury VFM Guidace. 3 Cotract Completio. This relates to all the activity betwee selectio of preferred bidder ad fiacial close 5. 4 Pre-Operatioal Implemetatio. This covers the phase betwee cotract close ad the start of operatioal services with a ew (or refurbished) asset. It covers procuremets where the costructio or delivery of ifrastructure is part of the cotract. 5 Early Operatioal. This relates to performace over the first three years after the start of operatioal services. Three years takes i the period of iitial beddig dow of the cotract ad the developmet of a log-term relatioship betwee both parties. 6 Mature Operatioal. This covers performace from the begiig of the fourth year i which services are operatioal util the ed of the cotract 6. Past NAO work idicates that six key busiess-maagemet themes come ito play at ay phase of a PFI project s lifecycle 1.8 Based o our discussios with private ad public sector represetatives ad by cosiderig our previous work, we were able to defie six key busiess-maagemet themes that apply at every phase of a PFI project: The project fits with the busiess requiremets of the Authority. The project desig should seek to be a optimal fit with the Authority s core busiess requiremets ad cotiue to deliver a optimal outcome. PFI is the appropriate delivery mechaism. The decisio to procure through ad cotiue with a PFI route must be clearly demostrated ad be cosidered better tha ay alteratives. Stakeholders support the project s progress. Relevat stakeholders should be egaged ad satisfied with the developmet of the project ad the Authority must maage stakeholder iterests appropriately. There is good quality project maagemet. The project maagemet structure should be desiged to esure that outcomes from each phase are optimal for the busiess. 4 HM Treasury: PFI: Stregtheig Log-term Parterships, (2006). 5 Stage 3 of the Treasury Guidace also covers this phase. 6 Experiece i this phase is limited to date, ad it is likely that our framework will eed to be ameded as more experiece ad evidece accumulates. 6 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

9 part oe There is a optimal balace betwee cost, quality ad flexibility. The Authority eeds to achieve ad maitai a good value deal which is affordable, meets service requiremets ad provides a fiacial structure allowig for flexibility i the evet of chagig busiess eeds. Effective risk allocatio ad maagemet is takig place. Risks eed to be placed with the party best able to deal with them ad proper cosideratio give to the trade-off betwee trasferrig risks ad the costs of doig so. 1.9 These six themes have bee ordered to reflect the process of costructig a effective PFI deal ad subsequetly receivig the required services. For example, good quality project maagemet is irrelevat if the project does ot have the support of relevat stakeholders. I tur, stakeholders should ot be egaged ad maaged if the Authority has ot, at the least, already demostrated that PFI is the appropriate delivery mechaism for a project that satisfactorily fits with its busiess requiremets. Furthermore, the full beefits of good project maagemet structures are oly realised if the Authority is able to secure a deal that is affordable ad has a good allocatio ad maagemet of risks. The idicators are arraged i a matrix laid out by project phase ad busiess theme 1.10 As all six busiess themes apply across all six project life-cycle phases, it is possible to set out the idicators i a matrix, as show i Figure 1 overleaf Private Fiace Uits also have a iterest i idividual PFI projects ad ca use the framework i the same way, but they also eed to review PFI programmes as a whole. The OGC Gateway Review process defies a ivestmet programme as a portfolio of projects that have certai commo characteristics ad which are selected or commissioed, plaed ad maaged i a co-ordiated way ad which together achieve a set of defied busiess objectives Programme evaluatio also eeds to take accout of issues icludig: whether lessos from earlier projects have bee applied to later oes to improve desig ad performace; whether the opportuities for ecoomies of scale across projects have bee exploited; ad whether the programme has bee plaed i a way that is sustaiable withi supply capacity Our framework is desiged to complemet guidace such as that issued by the Treasury. The 2004 Treasury VFM Guidace sets out a rage of requiremets withi the process, aimed at achievig good outcomes. Our framework provides a methodology for cosiderig whether good outcomes i the process of implemetatio have bee achieved. It remais importat however that official guidace is used fully to achieve well desiged ad good value for moey PFI projects i the first place. The matrix framework is iteded as a tool to assess PFI project implemetatio but is ot a replacemet for official guidace 1.11 Our framework ca be used i a umber of ways. Primarily it is iteded to set out the priciples which value for moey auditors such as the Natioal Audit Office ca apply. However, service maagers ad lead project officers ca also use it to moitor ad assess their ow idividual projects. 7 Office of Govermet Commerce: Gateway Review 0: Strategic Assessmet. A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

10 part oe 1 The Matrix Theme Phase Strategic Aalysis Tederig The project fits with the busiess requiremets of the Authority The best form of project to pursue has bee selected Top level specificatios for the required services have bee draw up The output specificatios i the teder properly address the busiess requiremets Robust paymet ad performacemeasuremet regimes are i place that clearly reflect optimal busiess requiremets The Authority is clear about its approach o balacig flexibility for chage agaist price for the proposed deal PFI is the appropriate delivery mechaism The project has bee assessed as beig part of a suitable ivestmet programme for PFI There is a good outlie busiess case justifyig a PFI procuremet route The qualitative reasos for proceedig with PFI are clearly justified The quatitative reasos for proceedig with PFI are clearly justified Service performace uder curret arragemets has bee baselied for future moitorig of PFI cotractor performace The optimal project structure for the Authority is acceptable to potetial private sector parters ad fuders The Authority has ecouraged iovatio by allowig a rage of solutios to be proposed The Authority has checked that the justificatio for a PFI solutio as cotaied i the busiess case still holds Stakeholders support the project s progress There is good-quality project maagemet A strategy is i place to commuicate with stakeholders o a ogoig basis Relevat stakeholder groups have bee cosulted i producig the outlie busiess case Goverace structures for the project have bee set up A realistic timetable for the procuremet of the project has bee laid out A well resourced ad experieced project team has bee put i place for the procuremet The required form of cosultatio with staff ad users durig the procuremet have bee idetified The seior maagemet iput required for a successful procuremet has bee idetified Key stakeholders have maitaied commitmet to the project A good project team is maitaied durig the tederig phase A clear ad realistic tederig timetable is i place ad maitaied Likely cotract issues have bee idetified before the start of tederig Procuremet costs have bee cotrolled A clear process for evaluatig bids ad settig assessmet criteria is i place The Authority is clear about the required goverace arragemets for the project oce it is operatioal A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

11 part oe Cotract Completio Pre-Operatioal Implemetatio Early Operatioal Mature Operatioal The project strategy ad likely outcome have bee re-evaluated to esure they are still i lie with busiess eeds The cotract meets the Authority s busiess requiremets The asset is beig delivered to cotractual specificatio Service-provisio outtur is meetig cotractual requiremets The asset is fit for purpose Where cotractual services differ from busiess requiremets, the Authority is actig to alig them Service-provisio outtur is meetig core busiess requiremets The asset if still fit for purpose ad is maitaied to a good stadard A review cofirms that alteratives were evaluated ad fairly elimiated The baseliig of service provisio has cotiued (to aid future assessmet of PFI service quality) [The Authority is sure that maitaiig the PFI route is the best optio i the evet of poor costructio outtur.] 1 The Authority has improved its performace sice the PFI asset became operatioal Maitaiig the PFI deal for future service provisio is assessed as the best value for moey Key stakeholders support the agreed deal Key stakeholders have bee kept iformed of project progress There is a good level of stakeholder satisfactio A good level of stakeholder satisfactio is beig maitaied Good procuremet maagemet has bee exercised durig the preferred bidder stage Appropriate cotractmaagemet arragemets are i place A good post-procuremet evaluatio has bee carried out All outstadig issues from cotract close are resolved There is provisio for effective oversight ad resolutio of material problems arisig Systems are developed to deal with the ew asset ad service provisio Goverace arragemets are still appropriate Correct skills trasfer withi the public sector has take place from procuremet to operatioal cotract maagemet The goverace structures are beig used appropriately Relatioships betwee Authority ad cotractor are workig well The Authority esures that the project team have appropriate skills ad kowledge for good service provisio The goverace arragemets are still appropriate Good ad costructive relatioships betwee both parties are beig maitaied The Authority takes steps to esure that the project team cotiue to have the appropriate skills ad kowledge for good service provisio [Has the Authority take steps to pla for the ed of the cotract] 2 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

12 part oe 1 The Matrix (cotiued) Theme Phase Strategic Aalysis Tederig There is a optimal balace betwee cost, quality ad flexibility Market soudigs have bee made to geerate maximum competitio for a PFI solutio The Authority ca be cofidet that the specificatio for the project will be affordable The optimal balace of the proposed project has ot bee compromised by favourable accoutig treatmet Quality bids addressig core busiess requiremets have bee received The ecoomically most advatageous bid has bee selected Effective risk allocatio ad maagemet is takig place The Authority has fully assessed risks associated with the project The Authority has maaged procuremet risks well Risk best maaged by the private sector is beig trasferred as part of the proposed cotract Procuremet risks have bee idetified ad risk-maagemet procedures put i place Source: Natioal Audit Office NOTES 1 This idicator is oly applicable i the evet of the asset beig deemed to be totally iappropriate for the Authority s busiess requiremets. 2 This idicator eed oly be cosidered whe the PFI cotract is earig its ed. 10 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

13 part oe Cotract Completio Pre-Operatioal Implemetatio Early Operatioal Mature Operatioal The deal is affordable i the short ad log term Competitive fiacig has bee achieved The fial agreed deal is the ecoomically most advatageous solutio. Price icreases oly relate to ew priority chages eeded by the public sector o terms that are good value for moey Affordability of the deal is beig maitaied Both parties are seekig to maximise quality Affordability of the deal is beig maitaied Both parties are seekig to maximise quality Periodic bechmarkig for price ad quality is takig place A review is coducted to esure that accoutig treatmet has ot compromised the deal s optimal balace The fial agreed risk trasfer cotaied i the cotract has bee reviewed for appropriateess The Authority has a risk-maagemet pla for use for whe the cotract goes live Risk-mitigatio procedures are workig properly The allocatio of risk has bee sustaied operatioally The risk implicatios of chages to the cotract are cosistet with the risk that was origially trasferred The Authority s riskmaagemet procedures are updated ad workig i lie with chagig circumstaces Risk trasfer remais optimal The Authority s riskmaagemet procedures are updated ad workig i lie with chagig circumstaces Accoutig treatmet for the asset remais cosistet with the actual risk trasferred to the private sector A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1 11

14 part two Part two Makig use of the matrix: the audit criteria This part provides a phase by phase guide to usig the matrix for evaluatig PFI projects at each mai poit i their life-cycle. Each of the six sectios i Part 2 addresses oe life-cycle stage, ad what would be expected for each of the busiess themes. This is best read as a summary which sets out the priorities. Evaluators ad others who wish to iform themselves of the detail ca refer to Aex 1 ad Volume Strategic Aalysis I this phase, aalogous to Stages 1 ad 2 of the 2004 Treasury VFM Guidace, the Authority should be producig a scope for the project ad the determiig whether seekig a PFI solutio is the best procuremet approach. The priorities for each of the themes durig Strategic Aalysis are: The project fits with the busiess requiremets of the Authority The Authority should have established a clear priority for the services beig procured through the project irrespective of procuremet route. The justificatio for the eed should be through a ivestmet appraisal which aswers key questios i accordace with the guidace i HM Treasury s Gree Book 8. The Authority should have draw up top level specificatios for the required services expressed i output terms. PFI is the appropriate delivery mechaism The Authority should idetify the most suitable procuremet route, firstly through assessig the project as part of a departmetal ivestmet programme ad the specifically through the completio of a Outlie Busiess Case (OBC). The OBC should clearly justify proceedig with the project through PFI for both qualitative ad quatitative reasos 9. The Authority should baselie curret service provisio to facilitate future realistic assessmet of a PFI cotractor s performace. The project structure has to be optimal for the Authority but withi the capability of the private sector to deliver the required outcome. This will require the private sector beig aware of the project ad the Authority s requiremets through market soudigs by the Authority. Stakeholders support the project s progress The Authority should have idetified potetial stakeholder groups, the diversity of their iterests ad at what stages the support of idividual groups are importat to a successful project. This requires a commuicatio strategy ad makig best use of feedback, icludig a reappraisal or stocktake o progress whe there are sigificat chages i stakeholder views. The OBC should be preceded by early cosultatio with key stakeholders icludig staff ad ed-users where there ca be beefits to the asset desig ad service specificatios of the project. 8 HM Treasury: The Gree Book: Appraisal ad Evaluatio i Cetral Govermet (2003). 9 The outlie busiess case should justify the legth of cocessio agreemet for the cotract ad also clearly demostrate why icludig soft service provisio is justified. These two issues are addressed i the recet HM Treasury publicatio: PFI: stregtheig log-term parterships (March 2006). 12 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

15 part two There is good quality project maagemet There should be a good structure for the subsequet PFI procuremet. This is maximised by a thorough aalysis of the optios for deliverig the project. The quatum ad timig of resource ad skills requiremets should be idetified. This should be liked to a well thought out procuremet timetable that ecourages competitive tesio. Key staffig (for which a dedicated resource is recommeded) ad roles o maagig the project should be idetified ad be i place at this early stage. There should be a appropriate goverace structure to cotrol the procuremet. Good project maagemet is ot optimised by a group operatig withi a silo. They have to ru the process for cosultig with stakeholders ad they eed the periodic commitmet of seior maagemet iput ad oversight. There is a optimal balace betwee cost, quality ad flexibility The Authority should be cofidet that the solutio is affordable based o a aalysis of the potetial costs of the project ad the likely fudig streams. Maximisig competitio by soudig out the market is the primary way to ehace value for moey. The project structure should be eutral of balace sheet accoutig treatmet. Effective risk allocatio ad maagemet is takig place The Authority should have idetified the scope for risk trasfer i advace ad settled o a optimum trasfer, allocatig risks to those best placed to maage them. There are a separate set of risks associated with proceedig with the project. These risks have to be idetified ad realistic procedures put i place to mitigate such risks if they occur. 2.2 Tederig This phase is equivalet to Stage 3 of the 2004 Treasury VFM Guidace. Based o a substatiated decisio to use PFI, the iteded outcome of the Tederig stage should be a preferred bid offerig the best possible value for moey. I evaluatig projects, evidece should be sought which idicates that the priorities for maximisig VFM were addressed: The project fits with the busiess requiremets of the Authority The busiess requiremets should have bee reflected i rigorous output specificatios, issued as part of the tederig documetatio to prospective bidders. This should be cosistet with robust paymet mechaism ad performace measuremet regimes that have already bee formulated ad are based o busiess requiremets. The documetatio should also iclude adequate procedures for maagig future chages to the Authority s requiremets. A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1 13

16 part two PFI is the appropriate delivery mechaism The Authority should have ecouraged iovatio ad the private sector has put forward a good rage of proposed solutios. The Authority should also have udertake a review of the project to eable it to coclude that the PFI solutio is either still worthwhile or take early corrective actio to put the procuremet back o track. Stakeholders support the project s progress The Authority should be maitaiig the commitmet of key stakeholders to the project, either through cosultatio i the productio of output specificatios or the approval of the most ecoomically advatageous bid. There is good quality project maagemet A good quality project team should have bee maitaied. The Authority eeds to have had assessmet criteria ad processes for evaluatig bids, should have idetified key cotractual issues ad idicated its thikig o possible goverace arragemets. These arragemets should have bee i place before ay ivitatio to teder. The Authority should have maitaied a clear ad realistic procuremet timetable with the mechaisms to cotrol procuremet costs. There is a optimal balace betwee cost, quality ad flexibility The Authority should have selected the most ecoomically advatageous bid from a rage of good quality bids that addresses its core busiess requiremets. Effective risk allocatio ad maagemet is takig place The risks best maaged by the private sector should have bee trasferred as part of the proposed cotract ad the Authority s risk maagemet pla updated where applicable. 2.3 Cotract Completio Oce the preferred bidder has bee selected, it is importat that the Authority maages the process up to fiacial close well. This is to esure VFM is optimised through both doig a rage of checks before sigig off ad also lookig further ahead by puttig i place arragemets for maagig the project oce the cotract has bee siged. Evaluatio issues at this stage are: The project fits with the busiess requiremets of the Authority The Authority should have esured that the project strategy ad proposed cotract reflected its curret requiremets. PFI is the appropriate delivery mechaism The Authority should have udertake a review which cofirmed that reasoable alteratives to the proposed deal were fairly evaluated ad elimiated; Stakeholders support the project s progress Stakeholders should have bee kept iformed of project progress with key stakeholders sigallig their agreemet i the fial busiess case. This icludes explicit support from ay bodies that provide the Authority with a sigificat stream of icome. There is good quality project maagemet Delays i gettig to cotract close should have bee miimised ad outstadig cotractual issues kept to a miimum. I additio the Authority should have its plas i place for future cotract maagemet. There is a optimal balace betwee cost, quality ad flexibility The Authority should have cofirmed that it has got the best trade off betwee quality ad affordability, esurig that the best fiacig terms available i the market have bee reached. There should have bee a separate review to cofirm that the accoutig treatmet does ot distort the value for moey. Effective risk allocatio ad maagemet is takig place The fial agreed risk trasfer should have bee reviewed for appropriateess ad the Authority should have had a risk maagemet pla to address either o-delivery of the plaed asset or service delivery failure i the operatioal phase. 14 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

17 part two 2.4 Pre-Operatioal Implemetatio The value for moey idicators for this phase of a project s life-cycle are cetred o esurig that the delivery of the asset is proceedig appropriately ad that the Authority is preparig for the start of the operatioal phase. The priorities at this stage are: The project fits with the busiess requiremets of the Authority The asset 10 should have bee delivered to time ad quality with ay icreases i the uitary paymet oly beig caused by iflatio uplifts ad cotract variatios. PFI is the appropriate delivery mechaism The baseliig of curret service performace should be cotiuig so that the Authority is i a strog positio to aalyse delivery uder the PFI procuremet route oce the asset becomes fully operatioal. Stakeholders support the project s progress Stakeholders should have bee kept iformed of progress i a way that is appropriate to their iterest i the project. There is good quality project maagemet The Authority should have bee able to resolve ay outstadig issues from cotract close as well as problems arisig durig the delivery of a ew asset. There should also have bee forward plaig for the operatioal phase i relatio to ew systems, cotract maagemet skills ad goverace arragemets. This is also the optimal time to have leart lessos from the procuremet ad its outcome through a post procuremet evaluatio. There is a optimal balace betwee cost, quality ad flexibility There should have oly bee agreed icreases i the uitary paymet where the Authority requested chages to specificatios. These chages should have bee assessed as good value for moey ad affordable. Effective risk allocatio ad maagemet is takig place The risk mitigatio procedures should be workig properly ad the risk maagemet pla should have bee updated durig this phase. 2.5 Early Operatioal We have defied this phase as lastig aroud three years from the commecemet of service with the ew asset (or trasfer of asset). It reflects our experiece of how log it takes before early operatioal problems are iroed out ad a costructive relatioship betwee private ad public sectors has bee fully developed. The priorities i this phase are: The project fits with the busiess requiremets of the Authority The services should have bee delivered to cotract based o a performace moitorig system ad paymet mechaism which are effective. The asset should be fit for purpose ad the cotractual services should be idetified as beig i lie with the curret busiess requiremets. PFI is the appropriate delivery mechaism The Authority should ot have to make a assessmet as to whether PFI is the appropriate delivery mechaism except where the costructio of the asset has resulted i a poor outtur. I those circumstaces the Authority should have assessed whether or ot it was appropriate to termiate the cotract. Stakeholders support the project s progress The Authority should have bee checkig that stakeholders are satisfied with how the asset ad associated services are performig. The Authority should also be takig steps to address cocers raised from cavassig of stakeholders. There is good quality project maagemet The Authority should have a cotract maagemet team that has the appropriate skills ad kowledge for the operatioal phase. The goverace structures should be used to proper effect ad the Authority ad cotractors should have a good workig relatioship. There is a optimal balace betwee cost, quality ad flexibility The balace betwee cost ad quality should have bee maitaied so that both are robust. Effective risk allocatio ad maagemet is takig place The Authority should have esured that service related risks trasferred uder the cotract were ot passed back to them ad that risk maagemet plas are beig updated. 10 Or, i some cases, the hadover of a asset that is to be used by the public sector but maitaied by the private sector. A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1 15

18 part two 2.6 Mature Operatioal As the operatioal period of the PFI cotract progresses, value for moey assessmet focuses o how well the deal is meetig the Authority s loger term strategic eeds. At a simple level, it is askig whether the deal has the flexibility to cope with chages i circumstaces that will occur over a period of 25 years or more ad still provide value for moey. The priorities i this phase are: The Project fits with the busiess requiremets of the Authority The Authority through regular moitorig of cotractor s performace should be updatig the cotract if ecessary to meet chagig busiess requiremets as well as checkig that the asset is maitaied ad fit for purpose. PFI is the appropriate delivery mechaism The Authority should be assessig whether the PFI deal has beefited the Authority ad cofirmig whether it will remai the best value for moey for future service provisio. Stakeholders support the project s progress The Authority should be esurig that stakeholder satisfactio is beig maitaied. There is good quality project maagemet The Authority should be maitaiig good project maagemet skills, a appropriate goverace structure ad good relatioships with the cotractor. It should also have made timely steps to pla for the ed of the cotract. There is a optimal balace betwee cost, quality ad flexibility The trade off betwee cost ad quality should have bee maitaied. Bechmarkig ad market testig exercises are oe test of this. Effective risk allocatio ad maagemet is takig place The allocatio of risk should remai optimal ad the accoutig treatmet remai cosistet with the actual risk trasferred to the private sector. Risk maagemet procedures should have bee updated to work i lie with ay chagig circumstaces. 16 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

19 aex oe ANNEX 1 Applyig the Matrix: Cosideratios for Evaluators 1 Strategic Aalysis Theme The project fits with the busiess requiremets of the Authority Idicators Has the best form of project to pursue bee selected? Have top level output specificatios for the required services bee draw up? Further questios Have clear objectives for the project bee set? Does the project meet policy imperatives? Was the project assessed as beig priority? Has a prelimiary evaluatio of the beefits sought bee made? Has log term commitmet to the project bee demostrated? Are the project outcomes clear? Have the project s wider socio-ecoomic beefits bee quatified? Does the proposed solutio clearly meet busiess requiremets? PFI is the appropriate delivery mechaism Has the project bee assessed as part of a suitable ivestmet programme for PFI? Has a good outlie busiess case justifyig a PFI procuremet route bee produced? Are the qualitative reasos for proceedig with PFI clearly justified? Ca a viable PFI cotract be costructed? Ca programme level objectives ad outputs be delivered by a PFI cotract? Is the Authority satisfied as to the operatioal flexibility of a PFI cotract? Is the Authority satisfied that there are o overridig reasos for providig the service directly? Is the private sector capable of deliverig the required outcome? A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1 17

20 aex oe Theme (cotiued) PFI is the appropriate delivery mechaism (cotiued) Idicators (cotiued) Are the qualitative reasos for proceedig with PFI clearly justified? (cotiued) Further questios (cotiued) Has it bee demostrated that PFI would brig sufficiet beefits? Is there sufficiet scope for the trasfer of risk to a private parter? Is there likely to be scope for iovatio? Is the trasfer of soft service provisio essetial for improved service delivery? Ca the service be assessed agaist a agreed stadard? Ca the PFI paymet mechaism icetivise the levels of service provided? Is the service suitable to be maaged o the basis of a log term cotractual relatioship? Is it possible to itegrate the desig, build ad operatio of the project? Is a PFI procuremet achievable? Is there capability to maage the project ad appraise ogoig performace agaist agreed outputs? Is there likely to be sufficiet market appetite for the project? Are the quatitative reasos for proceedig with PFI clearly justified? Is there a compariso betwee the likely costs of the PFI optio ad a public sector comparator (PSC)? Are the trasactio costs likely to be maageable? Is it based o soud evidece from all types of past procuremet experiece? Is the overall project affordable? Has sesitivity aalysis bee udertake to compute the effect of assumptios o the relative value for moey of the procuremet routes? Have the possible beefits as well as the costs of delivery optios bee quatified? Has service performace bee baselied for future moitorig of PFI cotractor performace? Is the optimal project structure for the Authority deemed acceptable for potetial private sector parters ad fuders? Stakeholders support the project s progress Is there a strategy to commuicate with stakeholders o a ogoig basis? Have the relevat stakeholder groups bee cosulted i producig the outlie busiess case? 18 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

21 aex oe Theme (cotiued) There is good quality project maagemet Idicators (cotiued) Have the goverace structures for the project procuremet bee set up? Further questios (cotiued) Has a realistic project procuremet timetable bee laid out? Has a well resourced ad experieced project team bee put i place for the procuremet? Do the project team members have the relevat skills? Is there appropriate legal, techical ad fiacial expertises? Is there appropriate procuremet expertise? Has the project team bee assembled i good time? Have adequate procedures bee put i place for maagig the risk associated with staff chages? Has a sufficiet amout of staff resource bee allocated to the project? Has the authority icorporated lessos leart from the previous procuremets? Has the form of staff ad user cosultatio required for the procuremet bee idetified? Has the seior maagemet iput required for a successful procuremet bee idetified? There is a optimal balace betwee cost, quality ad flexibility Have market soudigs bee take to geerate maximum competitio? Is the Authority cofidet that the specificatio for the project will be affordable? Has the optimal balace of the proposed project ot bee compromised by favourable accoutig treatmet for the Authority? Effective risk allocatio ad maagemet is takig place Has the project risk bee fully assessed? Are procuremet risks beig maaged well? Have the risks associated with the project bee idetified? Is the proposed risk allocatio sesible? Have the risks with proceedig with the project bee idetified? Has a mitigatig risk maagemet pla bee put i place ad bee followed? A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1 19

22 aex oe 2 Tederig Theme The project fits with the busiess requiremets of the Authority Idicators Do the output specificatios i the teder properly address the busiess requiremets? Have robust paymet ad performace measuremet regimes bee put i place that clearly reflect optimal busiess requiremets? Further questios Is the specificatio detail properly focused o outputs rather tha iputs? Is the specificatio cosistet with the expected project deliverables? Were Authority staff appropriately ivolved i the idetificatio of requiremets? Are the output specificatios clearly commuicated to bidders i the Tederig documets? Did the Authority discuss their requiremets with prospective bidders to esure that they were practical? Is a robust performace-measuremet regime proposed? Are performace measures well defied ad objective? Is there a efficiet divisio of resposibility for moitorig performace proposed? Is there a robust paymet regime proposed? Does the proposed regime ecourage the cotractor to deliver the service to the level expected? Is paymet depedet o satisfactory delivery of the required services? Are paymet deductios proposed for poor delivery of services? Is the paymet regime calibrated to focus o the most critical aspects of service delivery? PFI is the appropriate delivery mechaism Is the Authority clear about its approach o balacig flexibility for chage agaist price? Has the Authority ecouraged the private sector to put forward iovative solutios? Do the teder documets iclude proposed procedures for itroducig chages to the Authority s requiremets? Has the Authority decided what would be a acceptable cost trade-off for log-term operatioal flexibility? Desig Variats? Operatioal Process Variats? Fiacig Variats? Output Variats? Risk Trasfer Variats? Has the Authority checked that the justificatio for a PFI solutio as cotaied i the busiess case still holds? Stakeholders support the project s progress Have all key stakeholders maitaied commitmet to the project? 20 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

23 aex oe Theme (cotiued) There is good quality project maagemet Idicators (cotiued) Has a good project team bee maitaied for the tederig phase? Further questios (cotiued) Is there access to sufficiet expertise whe required? Are there team members with good egotiatig skills? Has a clear ad realistic timetable for tederig bee put i place ad maitaied? Have likely cotract issues bee idetified before the start of tederig? Have procuremet costs bee cotrolled? Have advisers bee appoited after competitio? Have realistic budgets bee set ad updated where appropriate? Have costs bee moitored ad maaged? Has a clear process for evaluatig bids ad settig assessmet criteria bee put i place? Have clear assessmet criteria bee set i advace of tederig ad commuicated to bidders? Is the weightig of evaluatio criteria i lie with busiess objectives? Have specialist sub-groups bee set up to evaluate the differet compoets of the bids, e.g. fiacial, quality of service, fiacial stability of bidders, risk trasfer ad desig quality? Has the importace of relatioships ad partership workig bee icorporated as a assessmet criterio? Is the decisio makig process trasparet? Are clear guidelies for liaisig with bidders draw up? Is the Authority clear about the goverace arragemets for the project oce it is operatioal? There is a optimal balace betwee cost, quality ad flexibility Have quality bids addressig core busiess requiremets bee received? Has the ecoomically most advatageous bid bee selected? Is the cost of the bid affordable? Does the wiig bidder have a good track record? Is the proposed desig suitable for busiess requiremets? Are the proposed operatioal solutios suitable for busiess requiremets? Is VFM beig achieved without workers terms ad coditios beig degraded? Effective risk allocatio ad maagemet is takig place Is the risk best maaged by the private sector beig trasferred as part of the proposed cotract? Are the procedures for maagig procuremet risk workig ad beig updated where applicable? A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1 21

24 aex oe 3 Cotract Completio Theme The project fits with the busiess requiremets of the Authority Idicators Has the project strategy ad likely outcome bee re-evaluated to esure they are still i lie with busiess eeds? Does the cotract reflect the Authority s busiess requiremets? Further questios Are the origial objectives for the project still valid? Does the proposed deal fit with the latest circumstaces? Does the cotract reflect the deal that has bee egotiated? Are hadover/termiatio arragemets clearly specified? Are there appropriate sactios/bouses to icetivise the cotractor? Are the performace measures i lie with the busiess requiremets? Are there appropriate provisios for dealig with chagig requiremets? PFI is the appropriate delivery mechasim Stakeholders support the project s progress There is good quality project maagemet Was a review carried out to cofirm that alteratives were evaluated ad fairly elimiated? Do key stakeholders support the agreed deal? Has good procuremet maagemet bee exercised durig the preferred bidder stage? Are there appropriate cotract maagemet arragemets i place? Were all reasoable alteratives examied if there was doubt about the value for moey of the PFI deal? Was there a clear overview cofirmig the desirability of proceedig with the best deal? Have the differeces betwee the wiig bid ad cotract award bee limited? Were delays i gettig to cotract close kept to a miimum? Have outstadig issues at cotract close bee kept to a miimum? Have resposibilities for moitorig the cotract bee established? Has the Authority esured that key staff moitorig the cotract have the required kowledge ad skills? Are there appropriate arragemets for feedback from Authority staff iterfacig with the project? Are there appropriate processes for resolvig day-to-day operatioal issues? 22 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1

25 aex oe Theme (cotiued) There is a optimal balace betwee cost, quality ad flexibility Idicators (cotiued) Is the deal affordable i the short ad log term? Has competitive fiacig bee achieved? Is the fial agreed deal the ecoomically most advatageous solutio? Further questios (cotiued) Are there firmly secured sources of fuds for ay capital ijectios ito the project by the Authority? Are there firmly secured sources of fuds for ogoig paymets? Has a fiacig competitio bee held? Has the optimal fiace structure bee achieved? Has the asset desig bee assessed as beig optimal for users? Have the service specificatios bee assessed as what is required by service users? Have the beefits ad costs of the proposed asset ad services bee quatified? Has a review bee coducted to esure that accoutig treatmet has ot compromised the deal s optimal balace? Effective risk allocatio ad maagemet is takig place Has the fial agreed risk trasfer cotaied i the cotract bee reviewed for appropriateess? Does the Authority have a risk maagemet pla for use whe the cotract goes live? A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1 23

26 aex oe 4 Pre-Operatioal Implemetatio Theme The project fits with the busiess requiremets of the Authority PFI is the appropriate delivery mechaism Stakeholders support the project s progress There is good quality project maagemet Idicators Is the asset beig delivered to cotractual specificatio? Has the baseliig of service provisio cotiued? Have key stakeholders bee kept iformed of project progress? Has a good postprocuremet evaluatio bee carried out Further questios Is the asset beig delivered to time? Is the asset beig delivered to quality? Is there ay icrease i the uitary paymet? Has the evaluatio cosidered if the staffig of the procuremet was appropriate? Was there evaluatio of the use of exteral advisers? Was there effective plaig ad maagemet of the procuremet? Has the evaluatio idetified deficiecies i the cotract that eed to be resolved? Have the lessos from such a evaluatio bee dissemiated for wider learig? Are all outstadig issues from cotract close resolved? Is there provisio for effective oversight ad resolutio of material problems arisig? Has a clear timetable for resolvig such issues bee produced? Has a clear project maagemet process bee put i place for esurig the issues are resolved? Has a process bee agreed betwee the Authority ad the cotractor for resolvig problems? Are the appropriate levels of maagemet staff committed to resolutio procedures? Does the Authority have a fall-back pla i place if there is a delay to the asset becomig operatioal? 24 A Framework for evaluatig the implemetatio of Private Fiace Iitiative projects: Volume 1