August Key Issues in Aggregating Indicators for Accountability Determinations under ESSA

Transcription

1 August 2016 Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA

2 THE COUNCIL OF CHIEF STATE SCHOOL OFFICERS The Cuncil f Chief State Schl Officers (CCSSO) is a nnpartisan, natinwide, nnprfit rganizatin f public fficials wh head departments f elementary and secndary educatin in the states, the District f Clumbia, the Department f Defense Educatin Activity, and five U.S. extra-state jurisdictins. CCSSO prvides leadership, advcacy, and technical assistance n majr educatinal issues. The Cuncil seeks member cnsensus n majr educatinal issues and expresses their views t civic and prfessinal rganizatins, federal agencies, Cngress, and the public. Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA Authr: Ryan Reyna, Senir Assciate at Educatin Strategy Grup COUNCIL OF CHIEF STATE SCHOOL OFFICERS Tny Evers (Wiscnsin), President Chris Minnich, Executive Directr One Massachusetts Avenue, NW, Suite 700 Washingtn, DC Phne (202) Fax (202) Cpyright 2016 by the Cuncil f Chief State Schl Officers, Washingtn, DC All rights reserved.

3 Cntents System Purpse...3 Type f Aggregatin...4 Weights Methd fr Calculatin Cmmunicatin f Rating(s) Cnclusin Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 1

4 With the passage f the Every Student Succeeds Act (ESSA), states have wide authrity t cnstruct a schl accuntability mdel that can best advance cllege- and career-ready utcmes in their unique cntext. The law requires each state t meaningfully differentiate the perfrmance f its schls n an annual basis, using a set f defined academic indicatrs (e.g., academic achievement, student grwth, graduatin rate, prgress in achieving English language prficiency) and requiring at least ne indicatr f schl quality r student success. In making the annual differentiatins, the state must give substantial weight t each f the academic indicatrs, and in the aggregate, much greater weight than it prvides t the schl quality and student success indicatr(s). The U.S. Department f Educatin s prpsed regulatins 1 further require a single summative rating verall fr the schl and that each indicatr receives a rating, with bth the summative and indicatr ratings each having at least three perfrmance levels. 2 Over the next year, ESSA requires that all states review and revise their state accuntability systems cnsistent with federal requirements and new state authrity. The Cuncil f Chief State Schl Officers (CCSSO) has wrked with states t reaffirm key principles and prduce a radmap and resurces t help guide this wrk. Thse dcuments define hw states can build high-quality, cllege- and careerready systems f schl accuntability and supprts, and meet ESSA requirements in that cntext. This paper is designed t g deeper n the ptins states have in aggregating indicatrs t make schl accuntability determinatins. Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA This paper is intended as a resurce fr state educatin agency (SEA) staff as they develp their new ESSA accuntability mdels. There is n single right way t aggregate multiple indicatrs within an accuntability system t prvide annual schl perfrmance classificatins; rather there are a variety f key decisins that a state must make t develp a system that best fits its visin fr accuntability. As such, multiple ptins are presented alng with additinal key questins t prepare state educatin leaders fr the critical task ahead. The fllwing presents an initial landscape f decisins and ptins fr states with regard t the aggregatin f indicatrs fr accuntability determinatins. There are five big decisin pints that states must cnsider: 1. System purpse 2. Type f aggregatin 3. Weights 4. Methd fr rating calculatin 5. Cmmunicatin f rating(s) 1 References are made thrughut this resurce t the U.S. Department f Educatin s prpsed ESSA regulatins. All references are cntingent upn finalizatin f the regulatins. 2 There are a number f additinal accuntability requirements in ESSA and the U.S. Department f Educatin s prpsed regulatins that are nt specifically discussed in this resurce. Fr example, the annual meaningful differentiatin f schls must be based n all indicatrs fr all students and each subgrup f students. This paper is largely silent n the specific inclusin f accuntability indicatrs, r the measures that make up thse indicatrs. States will need t cnsider the cnstructin f their accuntability system within the full cntext f the law and subsequent regulatins. 2

5 System Purpse As described in CCSSO s State Strategic Visin Guide, state educatin leaders shuld apprach the design f an accuntability system by starting with key questins f purpse, including: what utcmes are the state hping t achieve fr students? What needs t change t achieve thse utcmes? And, what infrmatin des the state need t infrm that change? State leaders als will need t cnsider hw the system helps fcus the actins f students, parents, teachers and administratrs. It is critical that the state engage with each f these grups t ensure that the purpse f the accuntability system meets the needs f thse mst affected by the system. 3 While accuntability systems are intended t measure the impact f schls n student learning, they must als: Establish transparency in schl perfrmance fr parents and plicy makers; Make determinatins fr additinal reward, supprt r cnsequence; Enable the cntinuus imprvement f teaching and learning in the schl. Each purpse is imprtant fr states t cnsider when designing an accuntability system. Hwever, states will als need t determine which f these purpses is first amng equals. The decisin regarding purpse will significantly influence the apprach a state takes in addressing the ther areas identified belw. It is paramunt that all accuntability systems reliably measure the impact that a schl has n its students, rather than the impact student characteristics have n schl perfrmance. Cnstructing a mdel that fails t d this will likely prvide inaccurate infrmatin t parents, misidentify schls in need f supprt, and dwnplay the exceptinal wrk f educatrs and administratrs in schls that are excelling with large ppulatins f traditinally underperfrming students. Key Questins When defining the state s visin fr the accuntability system, state leaders shuld cnsider the fllwing questins: What is the mst imprtant utcme fr the accuntability system tday? In five years? What parts f the current state system are driving the desired utcmes and what des the state want t change? Hw can the state accuntability system drive desired behavirs and instructinal practices? What prcess will the state use t engage stakehlders in defining the state s visin fr accuntability? Hw will that visin be cmmunicated bradly? 3 Fr mre infrmatin n stakehlder engagement, see CCSSO s guide t stakehlder utreach in the develpment and implementatin f ESSA plans: Stakehlder%20Engagement%20Guide%20FINAL.pdf. Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 3

6 Type f Aggregatin T make annual determinatins f schl perfrmance, states will need t aggregate the multiple indicatrs in their accuntability system. 4 There are fur main ways that states can aggregate the grup f indicatrs that make up the state s system fr annual schl determinatins: index, gal-based, matrix, and dashbard. While these appraches may seem distinct, sme states have blended these t fit their unique cntext. See belw fr examples f each methd f aggregatin, as well as examples f mixed state aggregatin methds, using the required ESSA indicatrs fr demnstratin purpses. Fr additinal examples f current and prpsed accuntability mdels that meet the requirements f ESSA, see CCSSO s Accuntability Mdels Matrix. In cnnectin t the aggregatin mdels presented belw, states may als need t include additinal business rules t address ESSA requirements such as inclusin f subgrup perfrmance in schl determinatins, participatin f at least 95 percent f students n the state assessment, and identificatin f cmprehensive and targeted supprt and imprvement schls. While nt discussed in significant detail in this resurce, it is imprtant t recgnize that there are additinal decisin pints that may need t be made prir t a state finalizing its indicatr aggregatin methdlgy. Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA Index Currently in use by many states, an index is the numerical aggregatin f perfrmance acrss the full spectrum f measures. The measures are ften assigned a weight accrding t plicy preferences. The index can be used t assign scres t grups f measures (i.e., perfrmance n all state assessments) and/r the system as a whle. Index systems typically calculate a single summative schl rating and cmmunicate the rating thrugh a number, label, letter, r symbl (see sectin belw n cmmunicatin fr a mre detailed descriptin). Using an index aggregatin, a state wuld select the bttm 5% f scres n the index t identify its cmprehensive supprt and imprvement schls. 5 States that emply an index system include (amng thers): Delaware, 6 Kentucky, Flrida, Luisiana, and West Virginia. The value f an index aggregatin is that it prvides clear infrmatin t the public abut the plicy value f each indicatr in the accuntability system. Additinal benefits, limitatins, and cnsideratins are listed belw. 4 This paper is silent n the specific measures that can fulfill the required ESSA indicatrs. Examples are prvided fr illustrative purpses nly t demnstrate hw aggregatin ccurs. The examples are nt intended as a specific recmmendatin fr the measures that a state shuld include in its system. Sme f the state examples f aggregatin are based n the state s ESEA waiver, and thus, d nt include all f the required ESSA indicatrs. 5 There are additinal requirements in ESSA and the prpsed regulatins fr identificatin f cmprehensive supprt and imprvement schls, such as any high schl with a graduatin rate belw 67 percent. 6 Nte that under the accuntability system apprved in its ESEA waiver renewal applicatin, Delaware did nt prvide a single summative rating n its schl reprt cards; rather it prvided individual ratings in fur different categries. The full aggregatin f indicatrs was nly used fr identificatin f Reward, Fcus, and Pririty schls, nt fr public cmmunicatin. This system was designed t meet the ESEA waiver requirements, and as such, des nt include certain required cmpnents f ESSA (i.e., prgress in meeting English language prficiency). 4

7 Benefits: Simple fr parents and the public t understand verall rating Can maximize differentiatin between schls and create clarity between the rating levels Limitatins: Can minimize transparency f perfrmance n individual measures May be difficult t weight the aggregatin f individual measures apprpriately Perfrmance expectatins may be behind-the-scenes 7 Cnsideratins: Establishing plicy weights requires plitical agreement regarding accuntability system values (i.e., imprtance f status vs. grwth) Plicy weights may nt match numerical weights in calculatin (see weights discussin belw) Perfrmance expectatins can be set fr individual metrics and/r verall, which can lead t unintended cnsequences if nt prperly aligned Perfrmance expectatins can be nrmative (i.e., tp 5% f schls) r criterin-based (i.e., >90% perfrmance r distance frm gal) Figure 1. Index Example Delaware 8 Area/Measures Weight Pints Academic Achievement 25% 125 Prficiency ELA 7.5% 37.5 Prficiency Math 7.5% 37.5 Prficiency Science 5% 25 Prficiency Scial Studies 5% 25 Grwth 45% 225 Grwth in ELA 22.5% Grwth in Math 22.5% On Track t Graduatin 20% 100 On Track in 9th Grade 5% 25 4-year Chrt Graduatin Rate 10% 50 5-year Chrt Graduatin Rate 3% 15 6-year Chrt Graduatin Rate 2% 10 Cllege and Career Preparatin 10% 50 Cllege and Career Preparatin 10% 50 Ttal 100% Depending n the structure, subgrup perfrmance may als be behind-the-scenes. 8 Delaware s accuntability system under the ESEA waiver: Centricity/Dmain/404/DSSF_RefGuide_15-16_ pdf Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 5

8 Gal-based Similar t Adequate Yearly Prgress (AYP), and still in use in a small number f states, the gal-based apprach t aggregatin establishes certain perfrmance benchmark(s) and determines schl perfrmance based n whether the schl met the benchmark(s) r hw clse it was t meeting the benchmark(s). In this apprach, the gals wuld directly cnnect t the lng-term and interim prgress gals required under ESSA, and culd be set in tw different ways. First, the state can establish a single cnsistent perfrmance expectatin fr all schls. The perfrmance expectatin can be determined in respnse t a plicy pririty (i.e., greater than 90% graduatin rate fr all high schls) r thrugh the use f statistical methds t determine the feasible gal range. In general, statistically based gals are desirable t thse set in an ad hc manner because they are mre actinable and defensible; hwever, statistically based gals can als lead t undesirable gals frm a plicy perspective (i.e., a schl imprving its graduatin rate frm 50 t 55% ver three years). State leaders will have t balance the dual pririties f establishing gals that are bth rigrus and attainable. Secnd, the state can establish gals fr accuntability determinatins based n perfrmance imprvement (i.e., year-t-year change) r subgrup perfrmance (i.e., gap clsure). Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA Schl determinatins result frm whether a schl met its gal(s) (1) n a set number f indicatrs (i.e., a schl that meets six ut f eight gals is identified as Met Expectatins ); r (2) a minimum perfrmance gate n specific indicatrs (i.e., 60% Prficiency in ELA and Math). Schls that d nt meet either f these threshlds are autmatically identified fr supprt and interventin. Gal-based systems typically calculate a single summative schl rating and cmmunicate the rating thrugh a label (i.e., Met, Appraching, Nt Met) r number (i.e., six ut f eight gals met). Using a gal-based system, a state wuld likely identify its cmprehensive supprt and imprvement schls based n the greatest number f gals missed and/r distance frm gals. State examples f a gal-based mdel include (amng thers): Tennessee (mix f gal and index), Califrnia (prpsed 9 ; mix f matrix and gal), and Vermnt (prpsed). The value f a gal-based mdel is that it can be used t directly align crss-sectr plicy gals t create cherence acrss the educatin system. Fr instance, the state culd align its gals fr students cllege and career readiness upn exiting high schl with its higher educatin attainment gals. Additinal benefits, limitatins and cnsideratins include are listed belw. Benefits: Clear expectatins f perfrmance fr all schls Has histrical precedence that is relevant fr stakehlders Can take advantage f new predictive analytic mdels t set gals Limitatins: May narrw fcus n imprvement t students near benchmark Requires multiple decisins abut perfrmance expectatins because gals are set fr multiple indicatrs Depending n number f gals t meet, may be difficult fr educatrs t fcus their effrts Requires resetting perfrmance expectatins as a result f test r cut scre changes 9 Nte that at time f writing, CA currently des nt plan t prvide a single summative rating fr its schls. If the state mves frward as such, that wuld technically place the mdel under the dashbard heading; hwever, the dashbard wuld prvide infrmatin abut schl perfrmance in a mix f matrix (at the individual indicatr level) and gal appraches. 6

9 Cnsideratins: Determinatins can be binary (i.e., met/nt met gal f 70%) r based n distance frm the gal (i.e., pints based n distance frm 70%). Predictive analytic mdels t set gals are under develpment and culd take this established idea int the next generatin f plicy making. Hwever, negative histrical cnntatins f AYP may discurage next-generatin mdels. Prpsed regulatins require at least three perfrmance levels, which increases the number f decisins abut the apprpriate level f perfrmance, and may lead t cnversatins abut whether and hw the gals differ fr different schls r student ppulatins. Figure 2. Gal-based Example Theretical Measure Perfrmance Gal Status Prficiency ELA 63% 60% Meets Prficiency Math 49% 60% Des Nt Meet Prficiency Science 91% 70% Exceeds Prgress in ELP 14% 50% Des Nt Meet Grwth ELA Abve Average Average Meets Grwth Math Abve Average Average Meets 4-year Grad 81% 83% Appraching 6-year Grad 86% 85% Meets Cllege and Career Readiness 41% 35% Meets Pstsecndary Enrllment 22% 25% Des Nt Meet Figure 3. Mix f Gal and Index Example Tennessee 10 Step 1: Did schl meet minimum Achievement and Grwth expectatins? Step 2: Assign pints based n: Did schl meet its targets? Hw did it perfrm relative t peers? What level f grwth demnstrated? Step 3: Assign pints based n s whether schl clsed identified gaps Step 4: Average Achievement and Gap clsure t get final determinatin NO - In Need f Imprvement; YES - Prceed t Step 2 Achievement status determinatin Gap clsure determinatin Final schl rating: Prgressing, Achieving, r Exemplary 10 Tennessee s accuntability system under the ESEA waiver: attachments/esea_flexibility_request_apprval_summary_2015.pdf. This system was designed t meet the ESEA waiver requirements, and as such, des nt include certain required cmpnents f ESSA (i.e., prgress in meeting English language prficiency). Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 7

10 Matrix Currently in use internatinally, by states in a number f teacher accuntability systems, and prpsed in at least ne state s new ESSA accuntability plan, the matrix mdel reviews schl perfrmance in tw (r mre) dmains, ften status and grwth (r imprvement). Determinatins are based n where the schl falls within each dmain. Practically, this is dne thrugh separate aggregatin f indicatrs in each dmain, and then pltting schl perfrmance n each axis (either in terms f relative perfrmance statewide r with respect t state perfrmance gals). Decisins must be made abut which specific measures fall int the dmains and whether ne dmain receives preference (r weighting ) when making determinatins. Matrix systems typically calculate a single summative schl rating, but cmmunicate the schl s perfrmance in the tw dmains directly (i.e., High Status, Average Grwth). Using a matrix mdel, a state culd identify its cmprehensive supprt and imprvement schls based n the 5% f schls in the bttm left crner f the matrix. Because this is a relative measure, it is likely that the bttm left wuld need t be reset every three years fr new identificatin f schls. The value f a matrix mdel is that it prvides parents, educatrs and the public mre nuanced infrmatin abut tw different avenues f schl perfrmance. Additinal benefits, limitatins and cnsideratins are listed belw. Benefits: Cmmunicates the plicy values f the system thrugh the schl rating (i.e., that status and grwth are equally imprtant) Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA Des nt cmbine dissimilar measures, thus skirting many f the accuntability weighting issues (discussed belw) Limitatins: Few schl accuntability examples currently exist s the mdel can be mre difficult t explain t stakehlders May be harder t establish federal schl classificatins f cmprehensive supprt and imprvement and targeted supprt and imprvement May require additinal aggregatin acrss indicatrs t incrprate subgrup perfrmance r prduce a single summative rating Cnsideratins: Must decide hw t incrprate specific indicatrs (r imprvement) int each dmain and whether t add additinal dmains, such as peer perfrmance cmparisns, t the mdel t further differentiate supprt and interventin Establishing perfrmance expectatins requires plitical agreement regarding accuntability system values (i.e., imprtance f status vs. grwth) Prpsed regulatins requiring each indicatr t have a perfrmance designatin f at least 3 levels can cmplicate the cmmunicatin f a matrix 8

11 Figure 4. Matrix Example Theretical Aggregatin f Indicatrs Status ELA Math Science 4-year Grad Rate On-track in 9th CCR Prgress ELA grwth Math grwth Prgress in EL Prficiency Extended year Grad Rate CCR Imprvement Status Abve Average Satisfactry Prmising Exemplary Average Cncern Satisfactry Prmising Belw Average Pririty Cncern Satisfactry Belw Abve Average Average Average Prgress Figure 5. Mix f Matrix and Gal Califrnia (prpsed) 11 Measure Status Change ELA Assessment Intermediate Imprved Math Assessment Lw Maintained English Learner Prficiency Very Lw Imprved 4-year Grad Rate Intermediate Declined Chrnic Absenteeism High Imprved Suspensin Rate & Lcal Climate Survey High Imprved Significantly Cllege & Career Readiness Intermediate Maintained Basics Met Implementatin f Academic Standards Nt Met fr One Year Parent Engagement Met Dashbard 12 This apprach des nt attempt t aggregate indicatrs int a single schl rating, rather it is designed t present actual schl perfrmance n each measure in a public frmat. In this way, a full picture f schl perfrmance is prvided, withut attempting t cmbine perfrmance acrss multiple measures. ESSA des require annual determinatins t identify lwperfrming schls fr supprt and interventin, s a state still needs t develp and publicize its prcess (likely ne f the three ptins described abve) fr making thse determinatins, 11 and dcuments/july16item2addendum.dc 12 This refers specifically t the aggregatin f accuntability indicatrs, nt t the idea f prviding a cmpelling, publicly transparent cllectin f schl perfrmance measures. All states under ESSA are required t prvide schl reprt cards that display infrmatin abut schl perfrmance acrss multiple indicatrs. States that use the ther aggregatin ptins and chse t display a single summative rating will still prvide a dashbard f data. Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 9

12 even if the infrmatin is nt publicly displayed thrugh a rating label. 13 This des nt preclude a state frm priritizing the display f perfrmance infrmatin thrugh a dashbard; rather, it means that the state will need t identify a specific set f business rules t identify schls fr the required federal classificatins. Illinis is an example f a state that previusly cmmunicated its accuntability infrmatin thrugh the use f a dashbard. The value f the dashbard mdel is that it aims t treat the accuntability system as a tl fr cntinuus imprvement while aviding the need t cmbine multiple measures int a single rating. Prviding insight t perfrmance n each measure can highlight gaps in even the highest perfrming schls. Additinal benefits, limitatins and cnsideratins are listed belw. Benefits: Maximizes transparency f perfrmance n individual measures Allws stakehlders t determine their wn values abut the data Des nt cmbine dissimilar measures, and thus culd skirt many f the accuntability weighting issues (discussed belw) Limitatins: Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA May be difficult fr the public t interpret verall perfrmance acrss schls Des nt prvide clear perfrmance expectatins fr educatrs and administratrs, thus may be challenging t understand why a specific schl was identified fr ne f the federal imprvement categries Effective cmmunicatin with dashbards takes cnsiderable design wrk Cnsideratins: Anther apprach needs t be used fr the identificatin f cmprehensive and targeted supprt and interventin schls 13 The current regulatins call fr at least three perfrmance categries. Under this rule, a state culd identify schls fr (1) cmprehensive supprt and imprvement, (2) targeted supprt and imprvement, and (3) all ther schls. ESSA des call fr meaningful annual determinatins and it wuld be up t interpretatin whether that apprach wuld meet the letter (r spirit) f the law. Additinally, the state wuld be required t publicize n its schl reprt cards if a schl has been identified in either the cmprehensive r targeted supprt and imprvement categry and the methd(s) used t identify thse schls. 10

13 Figure 6. Dashbard Example Illinis Illinis At-A-Glance Reprt Card JEFFERSON HIGH SCHOOL 4145 SAMUELSON RD ROCKFORD, IL (815) F A S T F A C T S 13.9% ready fr next level 61% graduatin rate 88% teacher retentin 1,836 students State Avg. State Avg. State Avg. Academic Success Grades: 9-12 District: ROCKFORD SD 205 All Illinis students in grades 3-8 take the PARCC assessment each year. High schl students take the PARCC in specific Math r English Language Arts (ELA) curses. Schl District State Success by Student Grup 35% 29% 22% 13% 1% 23% 30% 25% 20% 3% 14% 24% 28% 29% 4% Did Nt Meet Partially Met Apprached Met Exceeded READY FOR NEXT LEVEL This display shws PARCC perfrmance levels fr each student grup. N data is shwn fr grups with fewer than 10 students. White Black Hispanic Lw Incme English Learners With Disabilities Student Characteristics 33% 25% 24% 16% 2% 47% 30% 19% 4% 30% 32% 23% 14% 0% 34% 30% 23% 13% 0% 69% 31% 60% 23% 14% 4% Did Nt Meet Partially Met Apprached Met Exceeded READY FOR NEXT LEVEL White 32% Lw Incme 87% Black 25% English Learners 3% Hispanic 37% With Disabilities 16% Asian 3% Hmeless 7% American Indian 0% Tw r Mre Races 4% Pacific Islander 0% Schl Envirnment Effective Leaders Supprtive Envirnment Ambitius Instructin Fr mre infrmatin, visit IllinisReprtCard.cm Principal: Dn Rundall Superintendent: Dr.Ehren Jarrett Less Effective Leaders D principals and teachers implement a shared visin fr success? Average Cllabrative Teachers D teachers cllabrate t prmte prfessinal grwth? Less Ambitius Instructin Are the classes challenging and engaging? Less Supprtive Envirnment Is the schl safe, demanding, and supprtive? Less Invlved Families Des the entire staff build strng external relatinships? Cllege Readiness Cllabrative Teachers Invlved Families Ready fr Cllege Cursewrk Students wh meet r exceed ACT cllege readiness benchmarks Pstsecndary Enrllment Students wh enrll at clleges and universities Pstsecndary Remediatin (lwer is better) Students enrlled in Illinis cmmunity clleges wh require remedial cursewrk The 5Essentials Survey allws students in grades 6-12 and all teachers t share their perspectives n essential cnditins fr learning. The annymus survey cnsists f 5 cmpnents. Mst implementatin Mre implementatin Average implementatin Less implementatin Least implementatin Nt Applicable/Lw Respnse 19% 29% 46% Respnse Rates Students 55% Teachers 90% 0% 25% 50% 75% 100% 52% 56% 73% 49% 48% 49% Schl District State Key Questins: When deciding the apprach the state will take in aggregating accuntability indicatrs, state leaders shuld cnsider the fllwing questins: Hw can the state balance ease f cmmunicating the determinatin (r rating) with the transparency f perfrmance n individual indicatrs? What prcess des the state plan t use t establish the perfrmance expectatin cut pints? Des the state plan t establish perfrmance expectatins fr each indicatr r the whle system? What is an apprpriate level f perfrmance fr each designatin? What additinal measures beynd thse identified under ESSA will the state reprt t prvide a cmplete picture f schl perfrmance? Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 11

14 Weights Accuntability systems based n multiple measurements f perfrmance ften require the weighting f individual measures t calculate a summative rating. Under ESSA, states will be required t weight accuntability indicatrs t determine which schls t identify fr cmprehensive and targeted supprt and imprvement. ESSA calls fr the first fur academic accuntability indicatrs t accunt fr much greater weight than the fifth indicatr (e.g., schl quality r student success). Determining apprpriate weights fr each indicatr in the system is critical because it cmmunicates t educatrs and the public what type f perfrmance the state values mst. It is likely that decisins n indicatr weighting will require stakehlder engagement and ptentially plicy maker apprval beynd the SEA. It is imprtant t recgnize that the term weight in this scenari is almst always referring t the plicy weight assigned t a measure, which reflects system values (i.e., the state values grwth mre than prficiency) rather than the actual numerical weight f a measure. The ultimate value that a measure has in the verall calculatin f perfrmance is mre affected by the spread f perfrmance fr that measure than the weight assigned in the system. An example f this phenmenn is prvided belw in Figure 7. The figure displays a set f ptential high schl accuntability measures. Sme f the measures, like English language arts and math grwth, have a large spread r distributin f perfrmance, while thers, like 4-year graduatin rate and Ontrack t Graduatin, have relatively little spread. A state culd assign significant weight in its accuntability system t the 4-year graduatin rate and On-track t Graduatin measures (say 50%); hwever, because schl perfrmance is relatively clumped, very little differentiatin between schls wuld result. Rather, the measures with the greatest spread wuld mst significantly cntribute t the verall differentiatin f schl ratings. Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 12 Figure 7. Example f nn-standardized high schl accuntability measures. HS Metric Distributins (Nt Standardized) Percent f Ttal ELA grwth Math grwth year grad rate 5-year grad rate 6-year grad rate Cllege & Career Ready ELA prficiency Math prficiency Science prficiency On-track t Grad Index (0 t 100)

15 There is a slutin t avid this scenari, but it t can lead t cmplicatins with the calculatin and cmmunicatin f ratings at the individual measure level. States can standardize (e.g., a statistical prcess t create spread in metrics, as demnstrated in Figure 8) all f the measures in their accuntability system, s that the plicy weights chsen match the numerical weights in the verall calculatin. Standardizing makes perfrmance relative, and thus may frce schls t the bttm f perfrmance n a specific measure, even if the level f perfrmance is acceptable. Fr instance, a schl with a 70% 4-year graduatin rate may receive little t n credit (r pints) in the verall rating fr that perfrmance level if the measure is standardized and that is the lwest graduatin rate in the state. Hwever, this issue can als be crrected with additinal statistical techniques. Thus, it is imprtant t seek technical assistance frm agency data analysts, state Technical Advisry Cmmittees, r external research rganizatins n statistical weights regardless f the state s methd fr aggregatin r calculatin f perfrmance fr individual measure. 14 Further, it is imprtant t seriusly cnsider the value f additinal cmplicatins in the system. While standardizatin des allw fr mre accurate weighting amng measures, it can lead t mistrust amng stakehlders if they feel that the calculatins are t cmplicated r dne in a black bx. Cmmunicatin is essential in this area. Figure 8. Example f standardized high schl accuntability measures. HS Metric Distributins (Standardized) Percent f Ttal ELA grwth 4-year grad rate ELA prficiency Math grwth 5-year grad rate Math prficiency Index (0 t 100) 6-year grad rate Cllege & Career Ready Science prficiency On-track t Grad 14 Frm a technical standpint, states shuld als cmpare the verall schl ratings with the individual indicatr ratings t ensure that there is alignment Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 13

16 Key questins: When deciding hw t weight multiple indicatrs f perfrmance in the accuntability system, state leaders shuld cnsider the fllwing questins: Hw will the state determine the weighting f the multiple indicatrs in its accuntability system? Wh will be invlved in that prcess? Hw will the state ensure that the numerical weights f individual indicatrs match the plicy weights set frth by the state and its stakehlders? Hw will the state ensure that it meets the ESSA statutry language fr the academic indicatrs t cunt fr much greater weight than the schl quality r student success indicatr? What are the trade-ffs f using standardizatin t create a spread f perfrmance in indicatrs r crrect a perceptin caused by such a technique? Hw des that align with the state s gals arund transparent reprting? What capacity internal r external can the state rely n t cnduct apprpriate statistical tests n the accuntability system and its individual cmpnents? Methd fr Calculatin Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA There are five main methds currently in use fr calculating schl perfrmance n specific accuntability indicatrs. Sme states use a mix f the fllwing appraches, s they shuld nt be thught f as mutually exclusive Numerical In use in mst states, this methd f calculatin takes the perfrmance f a schl n a specific measure and cnverts it t a scale fr the mathematical aggregatin f multiple measures. Fr instance, 4-year graduatin rate is wrth 10 pints in the accuntability system, s a schl with 75% n that measure wuld receive 7.5 pints. Alternatively, the state may set perfrmance ranges (i.e., 70% and <80%) t attribute a certain amunt f pints. While this calculatin methd is mathematically transparent, it can als suffer frm issues related t differences in the scales f specific measures. Metric weights in particular may becme distrted withut careful cnsideratin. Nevada is an example f a state that emplys this methd Distance t gal In use in a few states, this calculates perfrmance based n hw clse r far a schl is frm the perfrmance expectatin fr a specific measure. Fr example, a state sets 4-year graduatin rate gal f 90%, s a schl with 75% graduatin rate wuld receive 8.3 pints (e.g., (75/90)*10=8.3). This methd requires setting a gal fr each measure and using numerical aggregatin nce perfrmance against that gal is weighted. Cnnecticut is an example f a state that emplys this methd Fr instance, Tennessee calculates perfrmance thrugh a number f methds, including: numerical, distance t gal, perfrmance against self and perfrmance against peer. Fr mre infrmatin, see: assets/entities/educatin/attachments/esea_flexibility_request_apprval_summary_2015.pdf

17 3. Perfrmance against self This methd f calculatin takes int accunt a schl s yeart-year imprvement. Fr instance, a schl with 75% graduatin rate in the current year and 70% graduatin rate in the prir year may receive full credit (i.e., 10 pints) n the measure fr demnstrating imprvement abve state expectatins r pints based n a sliding scale f imprvement (i.e., imprvement >4% = 10 pints, >3% and <4%=8 pints). The challenge with this methd is that the cmparisn ccurs between different chrts f students, s there is greater ptential fr year-t-year fluctuatins. The use f rlling, multi-year averages may be ne way t address this issue. Califrnia s prpsed mdel (shwn abve) prvides an example f this methd. 4. Perfrmance relative t peers This methd f calculatin seeks t evaluate perfrmance in relatin t similarly ppulated schls. Fr instance, take twenty schls with similar ppulatins f lw-incme students. Nineteen f the schls have a 50% graduatin rate and ne schl has a 60% graduatin rate. Even thugh all f the schls may be underperfrming state expectatins, the schl with the 60% graduatin rate wuld receive additinal credit fr utperfrming its peers (i.e., 10 pints vs. 5 pints fr schls at the average f peer perfrmance). The difficulty in implementing this apprach is establishing meaningful peer grups. New Yrk City previusly emplyed this methd Cnjunctive/Bnus In use in a few states, ften in additin t anther apprach, this methd f calculatin identifies specific measures that can mve a schl s rating up r dwn a designatin scale based n perfrmance. This methd is ften used t uphld r place significant emphasis n a specific accuntability value, such as greater than 95 percent f students testing. In the cnjunctive mdel, a schl may be required t meet a specific perfrmance threshld n a measure (r set f measures); therwise their rating may be predetermined r reduced. Fr instance, a schl may be required t meet the 95% assessment participatin threshld, and if they miss that target, they autmatically receive a lw schl designatin. A similar apprach can ccur t mve schls up the designatin scale. Often termed a bnus, a number f states prvide additinal credit t schls fr meeting certain expectatins n accuntability measures. Fr instance, a schl may earn up t an additinal 5 pints based n the percentage f students wh enrll directly in pstsecndary educatin r training after high schl graduatin. Gergia is an example f a state that uses this methd thrugh its Exceeding the Bar indicatrs HS_11_25_2013.pdf 19 and%20targets/2014%20ccrpi%20indicatrs% %20v2.pdf Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 15

18 Key Questins When deciding hw the state will calculate accuntability indicatrs, state leaders shuld cnsider the fllwing questins: What data, infrmatin systems, r technlgy supprts will the state need in place t deliver n its apprach t accuntability aggregatin? Hw shuld the SEA s chief infrmatin fficer r data management team be invlved in the cllectin, calculatin, and aggregatin f accuntability indicatrs? Which, if any, indicatrs are included t demnstrate an accuntability expectatin fr all schls (i.e., any measure f perfrmance that autmatically puts a schl in an imprvement categry r mves a schl up a designatin categry)? T what extent will the state use multiple years f data (r rlling averages) t calculate ratings r accunt fr imprvement in ratings? Hw will the state incrprate the 95% assessment participatin requirement in ESSA? Hw will the state incrprate the requirement in ESSA t include subgrup perfrmance in annual schl determinatins? Cmmunicatin f Rating(s) Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA States use a myriad f ways t display their accuntability determinatins. 20 Again, these tactics are nt necessarily mutually exclusive; many states include specific ratings fr bth verall determinatins and perfrmance in specific categries f measures. This apprach can increase public transparency. 1. Numbers This can take the frm f traditinal (ex , 1-5) r nn-traditinal (ex , 1-4) scales. Use f traditinal scales can be helpful fr public transparency as stakehlders ften have a frame f reference fr appraching the numbers. Hwever, that frame can create issues if perfrmance des nt fit the mdel as the public is used t experiencing it (fr instance, a scre f 70 ut f 100 may be the highest perfrming schl in a state, but wuld seem like a middle f the rad schl fr smene wh understands a 70 t reflect a C in schl grades). The inverse is true f nn-traditinal scales. Additinally, the use f large scales may lead stakehlders t interpret significant differences in schl perfrmance where they d nt actually exist. Fr instance, a 10-pint difference n a 1000-pint scale is unlikely t signal significant perfrmance differences. 2. Labels This can take the frm f state-determined language (i.e., belw expectatins, meet expectatins) r federally required categries (i.e., cmprehensive supprt and imprvement). Using wrds can limit schl-t-schl rankings. On the ther hand, the language used can make it difficult fr stakehlders t interpret hw well a schl is actually perfrming r hw clse a schl may be t the next perfrmance categry. 20 The prpsed regulatins call fr at least three levels f perfrmance fr the summative rating and fr each indicatr. Fr the summative rating, at a minimum, ESSA requires that states will identify and publicly cmmunicate schls designated fr (1) cmprehensive supprt and imprvement and (2) targeted supprt and imprvement. 16

19 3. Letter grades In use in a number f states, this places the designatin f schls n an A-F scale, similar t grades. This style f determinatin faces many f the same challenges as raised in bth the number and wrd appraches, as it is a specific cmbinatin f thse tw mdels. 4. Symbls This apprach takes a familiar symbl frm ther rating systems (fr instance, Frbes rates htels based n a quality rubric and cmmunicates the rating n a 5-star scale) and uses it fr schl designatins. This can prvide the public with a frame f reference fr reprting in a smewhat neutral way. Hwever, it may be difficult fr the public t understand hw the calculatin led t the specific rating, and the extent t which there is differentiatin between the levels. 5. Clrs Often used in cnjunctin with wrds r symbls, clrs are added t prvide an additinal level f cntext abut perfrmance t supprt the public s interpretatin f the results. Fr instance, a high perfrming schl may receive 5 green stars while a lw perfrming schl receives 1 red star. Questins arise abut the adequate number f clrs t include, and whether it is useful t g beynd the traditinal red-yellw-green stplight spectrum. Fr instance, sme states use blue t signify the highest level f perfrmance. 6. Dashbard This apprach pivts away frm prviding any specific ratings; rather, the gal is t prvide schl perfrmance data in its actual frm. While it is pssible t bth display individual perfrmance data and an accmpanying rating (and many states d), this apprach differs in that there is a purpseful chice nt t publicly label schls with statedefined rating(s). On ne hand, this apprach can encurage the use f the accuntability system as a cntinuus imprvement tl. On the ther hand, it can make it difficult fr the public t understand hw well a schl is perfrming. In this apprach, a state culd prvide cmparisn data (t ther schls, district r state) as a way f differentiating perfrmance fr the public withut prviding a specific rating fr each data element. Key questins: When deciding hw the state will cmmunicate accuntability determinatins, state leaders shuld cnsider the fllwing questins: Hw will the state engage with parents, educatrs, plicymakers and the public t gather feedback n the cmmunicatin f schl rating(s)? Which apprach t cmmunicatin d stakehlders find t be mst valuable? Hw many schl rating categries will the state use? What are the benefits and limitatins t prviding additinal nuance in the schl determinatins? What prcess will the state use t set the perfrmance threshlds? Wh will be invlved in that prcess? Hw much differentiatin exists between the state s chsen designatin categries? Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 17

20 Cnclusin Designing a system t aggregate multiple indicatrs is ne step in an nging prcess tward building a rbust accuntability system that prvides clear infrmatin t the public, empwers educatrs with data and infrmatin necessary fr imprvement, and targets supprts and interventins t the schls with the greatest needs. Thrughut the prcess, state leaders shuld cnsider hw the accuntability system culd evlve ver time, and establish systems fr peridic review and imprvement. It is paramunt that states priritize cmmunicatin with, and feedback frm, diverse stakehlders thrugh develpment, implementatin and future revisins. Accuntability can be a critical tl t supprt students n their path t cllege and career readiness. Hwever, accuntability lses value when students, parents, educatrs, and plicymakers struggle t understand what is measured, why it is measured, and hw it is cmmunicated. Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 18

21 This paper was written by Ryan Reyna, Senir Assciate at Educatin Strategy Grup and frmer Directr f Accuntability and Data Management fr the Delaware Department f Educatin, n behalf f the Cuncil f Chief State Schl Officers. They authr can be cntacted at rreyna@edstrategy.rg. The authr wishes t thank Katie Carrll and Sctt Nrtn at CCSSO fr their feedback, and special thanks t Andrew Rice at Educatin Analytics fr his supprt in the initial develpment f this paper. Key Issues in Aggregating Indicatrs fr Accuntability Determinatins under ESSA 19

22 One Massachusetts Avenue, NW, Suite 700 Washingtn, DC vice: fax: