The frmal review prcess relies n an evidence base cnstructed by: The Spnsring Organizatin (SO) under review, State level data Evidence cllected during the nsite prtin f the review Hierarchy f Evidence Taken tgether, this cllectin f evidence helps the review team determine whether each prgram apprval criterin is rated a cmmendatin, met, r finding in the final reprt. During the ffsite prtin f the review, the burden f evidence rests slely with the SO t prvide high-quality evidence n which decisins can be made. An SO rganizes and submits evidence by respnding t prmpts in ffsite dmain wrksheets. In all cases, evidence f impact is valued mre highly than evidence that describes inputs r plans (see: Hierarchy f Evidence graphic). The gal f this advisry is t supprt SOs understanding and self-reflectin f what high-quality evidence f impact lks like in the ffsite prtin f the review. Mre specifically, this advisry will prvide general guidance n hw t plan fr the incrpratin f internal (SO-cllected) and external (state-cllected) data int the frmal review. SOs prvide a cmprehensive bdy f evidence that may include quantitative evidence, data that can be cunted r quantified, and qualitative evidence, data that describes r characterizes attributes. With increased access t quantitative data n Edwin Analytics and public prfiles, SOs have many mre data pints t cnsider fr their cntinuus imprvement and accuntability. The flwchart belw prvides guidance t SOs t determine the sufficiency f the evidence prvided in the ffsite submissin. As the review prcess places the mst value n evidence f impact, the fcus f the flwchart is impact evidence. Nte: This tl is nt designed t be indicative f the verall rating f the ffsite submissin. Rather, it is intended t help SOs self-reflect n the strength and quality f the evidence f impact submitted fr review. Evidence statements that are written in alignment with these cmpnents will maximize the amunt f infrmatin that the review team is able cnsider in the decisin-making prcess. T further supprt SOs in incrprating evidence int the frmal review, this advisry includes several appendices. Appendix A includes additinal guidance abut the flwchart. Appendix B prvides a list f available state data and where t access them. Appendix C features additinal guidance arund the inclusin f quantitative data in the ffsite submissin. Appendix D prvides a self-assessment by dmain fr SOs t evaluate the quality f their evidence f impact in the ffsite submissin. SOs can use the tl t refine and finalize the ffsite submissin Tl Educatr Preparatin Review High-Quality Evidence f Impact 1
Sufficient evidence f impact in the ffsite prtin f the review has the fllwing cmpnents: High Quality Evidence f Impact Flwchart At a minimum, strng evidence f impact is: Outcmes-based Evidence fcuses n the results f the prgrammatic feature. Aligned with Criterin Practices and evidence f impact shuld be directly linked t the criterin being evaluated. It must address all aspects f the criterin. If yur evidence is utcmes-based and aligned with criterin, mve t the next cmpnents. Next, strng evidence f impact is: Specific, and wherever pssible, Evidence f impact shuld be specific abut the result fr each prgrammatic feature. It shuld be descriptive f the utcme and, when pssible, numeric in measure. Measured by multiple surces It is imprtant t triangulate evidence frm multiple surces, whenever pssible. SOs shuld cnsult the Review Evaluatin Tls in rder t see the multiple surces f evidence ESE uses t evaluate a single criterin. All evidence shared (whether it is state level data r SO-specific data) shuld be explained in a way that articulates the key takeaway(s). When pssible, it shuld: Capture multiple pints in time This culd include pre- and pst- data; frmative and summative data; and/r lngitudinal data frm multiple pints thrughut a year r ver multiple years. Overall, strng evidence f impact is: Cnnected t plans All evidence shared shuld be wven tgether in a cgent narrative that explains the cnnectin between evidence f impact and inputs and clearly articulates any subsequent plans. Educatr Preparatin Review High-Quality Evidence f Impact Tl 2
Appendix A: Examples The table belw prvides further guidance and examples f strng and weak evidence. Please nte, these examples are nt meant t be exhaustive. Guidance Example(s) Nn-Example(s) Outcmes-based What evidence tells yu this feature/initiative/ input is successful? What are the results? Student learning data Survey data Educatr evaluatin ratings Retentin data Qutes frm internal and external stakehlders explaining the impact they experienced after the prgrammatic feature/initiative/input Statements f activities Syllabi Meeting minutes Resumes/CVs Frequencies f meetings Aligned with Criterin Des the evidence speak specifically t what the criterin is measuring? whenever pssible, Is the evidence clearly indicative f the intended utcme/result? Measured by multiple surces D multiple surces f evidence crrbrate r reinfrce the claim? Are there discrepancies that need t be addressed? Uses language frm the criterin Addresses all cmpnents f criterin Survey data (see supplemental dc #1) indicates that 90% f candidates (n=67) said they were satisfied with the advising n licensure requirements they received in 2017. (Scale: Satisfied, smewhat satisfied, smewhat dissatisfied, dissatisfied) Multiple surveys SO-cllected data and state cllected data Data n the same utcme frm multiple stakehlder grups Quantitative and qualitative data pints related t the same utcme/result Discnnected r brad statements Des nt prvide reviewer with infrmatin f the extent t which the SO is meeting criterin Many candidates have expressed satisfactin with advising. Evidence f an utcme/result is limited t data frm a single data pint, single surce f evidence, r single stakehlder grup. Is the evidence written in a way that makes the cnclusin clear t the reader? Wuld multiple different readers reach the same cnclusin? pints in time Shares the data pint(s) and additinal cntext t explain the takeaway Pre-/pst- tests that measure grwth Graphical displays f prgress Frmative and summative data Shares the data pint(s) withut additinal cntext Single summative data pint Multiple data pints that d nt align with each ther Tl Educatr Preparatin Review High-Quality Evidence f Impact 3
Guidance Example(s) Nn-Example(s) Des the data articulate trends r patterns ver time? Cnnected t inputs and plans Are varius data pints clearly cnnected? Makes explicit cnnectin between data and practice Identifies the prblem and the impact f the slutin Tells the stry f data driven decisin making and resulting utcmes Lists f facts and/r data withut cnnecting it t the cntext r prmpt questin Tl Educatr Preparatin Review High-Quality Evidence f Impact 4
Appendix B: Data surces in Edwin The graphic belw indicates what data metrics are available in what Edwin reprt. Fr mre infrmatin abut the types f questins yu can answer in each reprt, see the Edwin Overview page. Educatr Preparatin Review High-Quality Evidence f Impact Tl 5
Appendix C: Sharing Quantitative Data in the Frmal Review ESE cllects and reprts n a number f utput data measures; ESE and reviewers use several f these measures t infrm decisins made in the frmal review fr prgram apprval. 1 SOs have the pprtunity t share state-cllected r SO-cllected data alng with additinal cntext, analysis, and their wn cnclusins based n the data t demnstrate the extent t which they are meeting ESE expectatins as utlined in the Prgram Apprval Criteria. Belw are general guidelines t cnsider while incrprating quantitative data int the ffsite prtin f the frmal review t ensure data included prvides the review team with the infrmatin they need t incrprate the data in their decisin making prcess: Guidance Ratinale Example Share data frm the three mst recent years Include the n-size If sharing survey data, include the entire survey scale fr the item in the prmpt respnse. 2 Althugh SOs may chse t include data frm any year, by including the last three years f data at a minimum, reviewers get an indicatin f the prgram in its mst current iteratin. This will help reviewers understand hw representative this data is f the ppulatin f stakehlders. It is imprtant fr reviewers t understand the full range f respnses in SO-cllected data. Candidates wh cmpleted between 2014 and 2016 agreed In 2014, we surveyed 8 cmpleters and fund that TCs are given Practicum Surveys exiting the prgram. 80% stated that they were very satisfied with their advising. (See supplement #1 Prgram Survey data) 1 See Quick Reference Guide Output Data (will be linked nce psted n website) 2 Nte: This will nt cunt against wrd limits. Educatr Preparatin Review High-Quality Evidence f Impact Tl 6
Appendix D: High-Quality Evidence f Impact Checklist Hw t use this checklist: When cnstructing yur ffsite submissin, use this checklist t help determine if yu are submitting high-quality evidence f impact. It may be beneficial fr smene wh did nt write the submissin t cmplete the checklist fr an bjective perspective. Organizatin: Is the rganizatin set up t supprt and sustain effective preparatin? Evidence f Impact is: ORG1 ORG2 ORG3 ORG4 ORG5 ORG6 Ntes/Examples Outcmes-based N Aligned with criterin N If yes fr the tw abve, cntinue t next sessins. If n, stp and recnsider the evidence yu are including. wherever pssible, N Measured by multiple surces pints in time Cnnected t plans N N N N Educatr Preparatin Review High-Quality Evidence f Impact Tl 7
Partnerships: Is the rganizatin meeting the needs f the PK-12 system? Evidence f Impact is: PAR1 PAR2 PAR3 PAR4 PAR5 Ntes/Examples Outcmes-based N Aligned with criterin N wherever pssible, Measured by multiple surces pints in time Cnnected t plans N N N N N Educatr Preparatin Review High-Quality Evidence f Impact Tl 8
Cntinuus Imprvement: Is the rganizatin engaging in cntinuus imprvement effrts that result in better prepared educatrs? Evidence f Impact is: CI1 CI2 CI3 CI4 Ntes/Examples Outcmes-based N Aligned with criterin N wherever pssible, Measured by multiple surces pints in time Cnnected t plans N N N N N Educatr Preparatin Review High-Quality Evidence f Impact Tl 9
Candidate: Is the candidate s experience in the prgram cntributing t effective preparatin? Evidence f Impact is: CAN1 CAN2 CAN3 CAN4 CAN5 Ntes/Examples Outcmes-based N Aligned with criterin N wherever pssible, Measured by multiple surces pints in time Cnnected t plans N N N N N Educatr Preparatin Review High-Quality Evidence f Impact Tl 10
Field-Based Experiences: D candidates have the necessary experiences in the field t be ready fr the licensure rle? Evidence f Impact is: FBE1 FBE2 FBE3 FBE4 FBE5 FBE6 Ntes/Examples Outcmes-based N Aligned with criterin N If yes fr the tw abve, cntinue t next sessins. If n, stp and recnsider the evidence yu are including. wherever pssible, N Measured by multiple surces pints in time Cnnected t plans N N N N Educatr Preparatin Review High-Quality Evidence f Impact Tl 11
Field-Based Experiences (cnt d) Evidence f Impact is: FBE7 FBE8 FBE9 FBE10 FBE11 FBE12 Ntes/Examples Outcmes-based N Aligned with criterin N If yes fr the tw abve, cntinue t next sessins. If n, stp and recnsider the evidence yu are including. wherever pssible, N Measured by multiple surces pints in time Cnnected t plans N N N N Educatr Preparatin Review High-Quality Evidence f Impact Tl 12
Instructin: D candidates have the necessary knwledge and skills t be effective? Evidence f Impact is: INS1 INS2 INS3 INSa INSb INSc Ntes/Examples Outcmes-based N Aligned with criterin N If yes fr the tw abve, cntinue t next sessins. If n, stp and recnsider the evidence yu are including. wherever pssible, N Measured by multiple surces pints in time Cnnected t plans N N N N Educatr Preparatin Review High-Quality Evidence f Impact Tl 13
Instructin (cnt d) Evidence f Impact is: INSd INSe INSf INSg Ntes/Examples Outcmes-based N Aligned with criterin N wherever pssible, Measured by multiple surces pints in time Cnnected t plans N N N N N Educatr Preparatin Review High-Quality Evidence f Impact Tl 14