Cmmissin d évaluatin de l enseignement cllégial Analytical Framewrk Evaluating the Effectiveness f Strategic Plans in the Cégeps April 2008
Dcument prepared and written by Katie Bérubé, Prject Crdinatr Secretariat: Danielle Buchard, Chantal Marcux Dcument available n the Cmmissin website at: http://www.ceec.guv.qc.ca The text f this dcument was adpted in its riginal French versin, Évaluatin de l efficacité des plans stratégiques des cégeps Cadre d analyse, by the Cmmissin d'évaluatin de l'enseignement cllégial at its 194 th meeting n April 4, 2008 Legal depsit 2008 Biblithèque natinale du Québec Natinal Library f Canada ISBN: 978-2-550-53089-3 (print versin) 978-2-550-53090-9 (PDF) Guvernement du Québec
Table f Cntents Intrductin... 5 The self-evaluatin prcess... 7 A. Expectatins f the Cmmissin... 7 B. Methdlgy... 7 C. Analysis f data... 8 Evaluatin f the effectiveness f the plans...9 A. Achievement f plan bjectives... 9 B. Critical analyses f the plans...9 Develpment cntext... 9 Mnitring f plans... 11 C. Overall assessment... 11 D. Fllw-up plans... 11
This dcument describes the Analytical Framewrk n which the expectatins f the Cmmissin are based and identifies the elements t be taken int cnsideratin fr the Cmmissin s analysis f the reprts frm the clleges n their evaluatins f the effectiveness f their strategic plans. Intrductin Changes made in December 2002 t the General and Vcatinal clleges Act and the Act respecting the Cmmissin d évaluatin de l enseignement cllégial required all cégeps t adpt multi-year strategic plans, including student success plans, and submit them t the Cmmissin fr evaluatin. The changes als made it mandatry fr the clleges t hencefrth make updates n prgress tward meeting the bjectives in their strategic plans an integral part f their annual reprts. These new legislative prvisins were designed t cntribute t the gvernment s gal f imprving services t the general public and mdernizing Québec public administratin, with special emphasis n results-based management practices, applicatin f the principle f transparency and increased accuntability. The develpment f strategic plans meant that the clleges, after analyzing their wn particular situatins and the issues assciated arising frm them, were in an excellent psitin t chse rientatins that best reflected their pririties and s prpse crrespnding strategic bjectives. In the Analytical Framewrk fr strategic plans sent ut t the clleges in February 2005, the Cmmissin stated that it planned t mnitr the implementatin f the strategic plans, in particular thrugh an evaluatin f their effectiveness. This evaluatin is ne part f a cntinuum f actins carried ut by the Cmmissin designed t cntribute t the nging imprvement f the quality, credibility and recgnitin f the imprtance f cllege-level educatin in Québec. The evaluatin prcess enables the Cmmissin t bth mnitr the implementatin f strategic plans in the clleges and pursue its evaluatin f the effectiveness f the varius quality assurance cmpnents (institutinal prgram evaluatin plicies, institutinal plicies n the evaluatin f student achievement and the clleges strategic plans, including student success plans) stipulated in the regulatins as being fundamental t ensuring the quality f educatin given by all institutins in the system. The evaluatin f cllege strategic plans is a tl whereby the Cmmissin can help clleges imprve their planning prcesses while als cnstituting a crucial reference fr them during the develpment f subsequent plans. 5
As it has dne in the past, the Cmmissin established an advisry cmmittee in the fall f 2007 t wrk with it during the peratin fr evaluating the effectiveness f the strategic plans. The cmmittee was cnsulted n bth the Guidelines and the Framewrk f Analysis. In the Guidelines dcument 1 sent ut in January 2008, the Cmmissin specified that each cégep was expected t chse its wn prcess fr evaluating the effectiveness f its strategic plan and assessing results, in relatin t its initial bjectives, while at the same time carrying ut a critical analysis f its particular plan. This wuld give the cllege the pprtunity t review its situatin by assessing the prcesses used fr the develpment, implementatin, mnitring and updating f the strategic plan, in additin t presenting its wn analysis f its strng pints, and f areas where imprvements t its strategic planning shuld be made. The evaluatin reprt t be prepared by the Cmmissin will cnsider the quality f each cllege s evaluatin prcess; it will appraise the effectiveness f the institutin s strategic plan, review the extent t which bjectives have been achieved, and give its assessment f areas r prcesses where the strengths f the cllege lie and f thse needing further attentin in the subsequent strategic plan. 1. Cmmissin d évaluatin de l enseignement cllégial, Evaluating the effectiveness f strategic plans in the cégeps Guidelines December 2007. (http://www.ceec.guv.qc.ca/publicatins/orientation-doc/effecaciteps_anglais.pdf). 6
The self-evaluatin prcess Each cllege shuld determine the prcesses and methdlgy t be used fr this peratin, in reference t its wn particular cntext. These prcesses shuld allw it t assess the effectiveness f its strategic plan, including its student success plan, thrugh a review f target bjectives and the results btained and thrugh a critical analysis f the plan as a whle. Reprts by clleges t the Cmmissin shuld describe the prcesses used in carrying ut the self-evaluatin. A. Expectatins f the Cmmissin 1. Each cllege shuld prepare a self-evaluatin plan identifying issues t be addressed in the evaluatin f the effectiveness f its strategic plan and specifying hw the evaluatin prcess will be carried ut. 2. Clleges shuld evaluate the effectiveness f their strategic plans, lking in particular at target bjectives and the results achieved. 3. Clleges shuld carry ut a critical review f the effectiveness f their plans, in terms f the cntext f their develpment and implementatin, the mnitring and revisin f their plans and any updating carried ut. 4. Each cllege shuld make an verall appraisal f its plan, identify where its strengths lie and where imprvements need t be made in its strategic planning, and then prepare a fllw-up actin plan. 5. Each cllege shuld cnsult the apprpriate rganizatinal structures and grups cncerned, bth as regards the self-evaluatin plan and the institutin s selfevaluatin reprt r any aspects f the reprt that invlve them. B. Methdlgy 1. Clleges shuld make use f the mst apprpriate indicatrs available fr assessing the achievement f target bjectives, whether r nt these indicatrs are identified in their plans. 2. Clleges shuld cllect relevant and sufficient data fr their analyses. 7
3. Clleges shuld be able t justify their chice f the methdlgy used fr the prcess. 4. Clleges shuld ensure that all apprpriate prcedures are in place t guarantee the cnfidentiality f infrmatin cllected, as required by the Act respecting Access t dcuments held by public bdies and the Prtectin f persnal infrmatin. C. Analysis f data 1. Clleges shuld be rigrus in carrying ut the analysis f the data they cllect. 2. Cnclusins that clleges arrive at shuld be relevant and based n the data cllected and analyzed. 8
Evaluatin f the effectiveness f the plans Clleges shuld evaluate the effectiveness f their strategic plans, including their student success plans, lking in particular at target bjectives and the results achieved, as well as carrying ut a critical analysis f the effectiveness f their plans as a whle A. Achievement f plan bjectives Fr all the target areas they have identified, clleges shuld present the results btained and, where apprpriate, explain the reasns fr any differences between these results and the initial bjectives. Clleges shuld take int cnsideratin bjectives adpted in their strategic plans, including bjectives fr student success and: Evaluate, with supprting arguments, the degree t which bjectives have been achieved; Explain any differences identified between the initial bjectives and the final results. B. Critical analyses f the plans Clleges shuld carry ut critical analyses f their plans and assess results btained, with reference t their plan develpment, implementatin and mnitring prcesses. Develpment cntext Each cllege shuld make a well-funded assessment f the analysis made f its wn particular situatin at the time the strategic plan was develped. The cllege shuld be able t shw whether it cnsiders that: The analysis f the situatin was cmplete; The results f the analysis enabled it t better understand its wn situatin, where apprpriate, and that the exercise was a useful tl fr mbilising the peple invlved; 9
The results f the analysis enabled it t identify issues pertaining t its particular situatin 2 ; fr the student success plan, the analysis helped the cllege better understand the characteristics f its clienteles and identify barriers t their success. Clleges shuld present a well-funded assessment f the prcesses used fr the develpment f their strategic plans. They shuld be able t demnstrate that: Target areas fr actin and the challenges represented by them were identified as a result f the analysis f their situatin. In the case f their student success plans, the identificatin f areas fr actin was based n their awareness f the characteristics f their different clienteles and bstacles t their success; Target bjectives were realistic and defined in ways that allwed the institutin t demnstrate the degree t which they were achieved. Plan implementatin Clleges shuld present well-funded evaluatins f the implementatin f their plans. They shuld be able t demnstrate that: The chsen strategies were implemented and resulted in the desired utcmes being achieved; Peple r grups required t intervene in the prcess carried ut their respnsibilities seriusly; Timelines fr the prcess were respected; The indicatrs selected were valid instruments fr mnitring prgress tward the achievement f the desired results. Making reference t issues identified during the evaluatin f the effectiveness f their plans, clleges shuld als assess the efficiency f the different means used t put the plans int effect and be able t justify the chices made. 2. That is, the cllege was able t identify areas f cncern and issues r challenges it faced. 10
Mnitring f plans Clleges shuld present a well-funded evaluatin f the effectiveness f the plan mnitring strategies used and shw that the mnitring prcess made it pssible t make any needed adjustments. C. Overall assessment In cnclusin t their strategic plan evaluatins, clleges shuld present an verall assessment f their plans and be able t demnstrate that these plans are effective. They shuld als be able t cnclude that their student success plans effectively led t successful results, perseverance in studies and success in btaining diplmas. Clleges shuld ultimately be able t shw that the plan develpment, implementatin and mnitring strategies they emplyed cntributed t the effectiveness f the plans themselves. D. Fllw-up plans All clleges shuld draw up fllw-up plans identifying needed imprvements, respnsibilities t be taken and timelines fr implementatin. The fllw-up prcess is a further way t increase the effectiveness f strategic planning in the clleges. 11