Staying On Track with Growth-to-Standards for English Learner Students Joni Lakin, Ph.D. Auburn University
Growth models poised to maintain or increase relevance under new assessment systems Many states have policies of status-plusgrowth accountability models Need to evaluate model performance for key subgroups
Same issues as status-based models: Heterogeneity Identification and reclassification Test Validity New issues: Data matching Changes in accommodations Low baseline scores Consistency and validity of models 3 6/5/2014
4 Compared behavior of four growth-based accountability models Value Table Model Trajectory Model Projection Model Student Growth Percentile Model Are EL students classified similarly to non-el students by each model? Are the classifications similarly accurate? Does the stringency of the proficiency cut score impact these results? 6/5/2014
Grade 6 is set as target (status-only) year Grade 3 (post-test) is first growth-eligible year Current plus one prior year of scores for SGP and Projection Designate students as on-track to future proficiency 5
Value Table: Positive transition made between proficiency categories Trajectory: Meet growth targets set by distance from current score to proficient score
Projection: Regression model predicts future scores above proficiency cut off in target year Student Growth Percentiles: Student s relative growth exceeds the rate needed to reach proficient by target year Images: Betebenner (2011). Retrieved March 29, 2012, from http://ccsso.confex.com/ccsso/2011/ webprogram/session2199.html http://scholar.harvard.edu/files/andrewho/files/a_pracitioners_guide_to_growth_models.pdf
Using 2015-2016 cohorts, grades pre-3 to 6 English Language Arts (ELA) Tests Ethnicity Non-ELs (207k) ELs (15k) White 59% 3% Hispanic 5% 83% African American 29% 2% Asian 2% 10% Student with disabilities 13% 10% Free or reduced lunch 45% 82% Migrant 0.1% 2% 8 6/5/2014
Cut score (z scale) Non EL Proficient EL Proficient Similar to 0.5 (low) 65% 33% >>State reports >>Hoffer et al. 0.0 54% 18% 0.5 (stringent) 31% 4% >>NAEP results >>Lakin & Young (CA data) 9 6/5/2014
Percent not proficient, but on track (out of all non-proficient) 50% 40% 30% 20% 10% 2016 cohort 46% 40% 31% 32% 23% 27% 25% 23% 20% 20% 17% 20% 14% 15% 15% 12% 16% 12% 14% 13% 15% 10% 10% 9% 0% grade 3 grade 4 grade 5
2016 cohort Percent not proficient, but on track (out of all non-proficient) 50% 40% 30% 20% 10% 0% 34% 28% 22% 33% 44% 37% 20% 16% 13% 12% 10% 7% 7% 5% 7% 7% 8% 5% 6% 4% 3% 0% 1% 1% grade 3 grade 4 grade 5
2016 cohort Percent not proficient, but on track (out of all non-proficient) 50% 40% 30% 20% 10% 0% 44% 37% 34% 33% 28% 22% 20% 16% 13% 8% 12% 7% 5% 7% 6% 6% 5% 5% 5% 0% 1% 2% 1% 3% grade 3 grade 4 grade 5
2016 cohort Grade 3 on-track classification vs. Grade 6 proficiency 100% 80% 60% 40% 20% 0% 82% 86% 86% 89% 86% 76% 79% 71% 72% 65% 27% 15% 14% 13% 10% 13% 8% 14% 14% 11% 11% 5% 8% 7% 7% 7% 5% 7% 8% 6% Status only VT Traj Proj SGP
2016 cohort Grade 3 on-track classification vs. Grade 6 proficiency 100% 80% 60% 77% 73% 76% 77% 64% 66% 81% 80% 76% 76% 40% 20% 20% 10% 26% 24% 14% 9% 15% 9% 15% 9% 7% 9% 13% 10% 11% 14% 8% 10% 10% 9% 0% Status only VT Traj Proj SGP
2016 cohort Grade 3 on-track classification vs. Grade 6 proficiency 100% 80% 76% 90% 67% 69% 66% 69% 90% 84% 78% 78% 60% 40% 20% 0% 27% 28% 16% 25% 27% 14% 9% 8% 7% 8% 10% 11% 6% 4% 4% 3% 6% 4% 6% 10% Status only VT Traj Proj SGP
VT+ TRAJ VT+ PROJ VT+ SGP TRAJ+ PROJ TRAJ +SGP PROJ +SGP On track by both models 22% 0% 6% 0% 7% 0% On track by ONE model 6% 37% 21% 38% 18% 19% Not on track by both 71% 63% 74% 62% 74% 81% On track agreement (row 1/1+2) 78% 0% 22% 0% 28% 0%
Average 6th grade gain (in SDs) given 3rd grade classification SDs (Cohen's d) 35 30 25 20 15 10 5 0 7 18 16 18 32 28 27 18 18 14 11 12 9 10 6 6 7 7 Status VT Traj Proj SGP Non EL Not on track Non EL On track Non EL Proficient EL Not on track EL On track EL Proficient
If goal is to identify students who will meet proficient cut-score in the future Growth models show little improvement over a naïve status-only model For stricter cutoffs, growth models worse than nothing predictively If goal is to identify students making larger gains Some success for all models Larger effects for Projection and SGP On-track determinations NOT interchangeable
Overall results Value Table and Trajectory optimistic, especially for ELs, less accurate at high cutoff Projection and SGP pessimistic, but accurate ELs on track make bigger gains than nonels Percent of proficient students DOES impact model performance Lower cutoffs result in more false negatives for ELs for all models, lower accuracy Although ELs are less likely to be on track with projection and SGP, those that are show large gains by 6 th grade
Lakin & Young found much smaller differences in % on track and accuracy of models for ELs CA district had large Hispanic population (50%) Here, just 5% ELs Proficiency rate close to L&Y(high cut score) led to pessimistic projection model observed in CA data, replicated here Lower cut scores in this study led to more similarity in the number of students ontrack across ELs and non-els
Further research Do growth classifications predict real achievement outcomes? Do they help plan changes to instruction? Explore differences in growth trajectories across groups Monitor match rates for ELs Track differential accuracy by key subgroups 21 6/5/2014
Confidential and Proprietary. Copyright 2011 Educational Testing Service. All rights reserved. 6/5/2014