Theory vs Practice of Mastery Learning in the Cognitive Tutor: Principal Stratification on a Latent Variable

Transcription

1 Theory vs Practice of Mastery Learning in the Cognitive Tutor: Principal Stratification on a Latent Variable Adam C Sales John F Pane University of Texas College of Education RAND Corporation SREE 3/1/2018 Adam Sales Latent Principal Strata 1 / 27

2 Outline 1 How Important is Mastery Learning? 2 Principal Stratification: Potential Mastery 3 We Need a Measurement Model! 4 Results/Model Checking 5 Discussion Adam Sales Latent Principal Strata 2 / 27

3 How Important is Mastery Learning? Intelligent Tutors and Mastery Learning Problem: Classroom full of students of diverse abilities Who to teach to? Students struggle on different things Adam Sales Latent Principal Strata 3 / 27

4 How Important is Mastery Learning? Intelligent Tutors and Mastery Learning Problem: Classroom full of students of diverse abilities Who to teach to? Students struggle on different things Solution: Students work individually on computerized intelligent tutors Adaptive: 1 Student works a problem skills 2 Software calculates P = Pr(Mastered Skills Work so far) 3 If P higher than a threshold, go to next section Mastery Learning Adam Sales Latent Principal Strata 3 / 27

5 How Important is Mastery Learning? Cognitive Tutor Algebra I One of the pioneers Seems to work... sometimes RAND effectiveness trial: Adam Sales Latent Principal Strata 4 / 27

6 How Important is Mastery Learning? Even in CTAI Mastery Learning Doesn t Always Happen Curriculum organized into sections, each containing several skills. In theory: graduate to next section iff master all of its skills In practice: Student exhausts all available problems, gets promoted Teacher moves student to other section Student stops CTAI work altogether Final Reassigned Promoted Mastered 81% 95% Adam Sales Latent Principal Strata 5 / 27

7 How Important is Mastery Learning? Even in CTAI Mastery Learning Doesn t Always Happen Curriculum organized into sections, each containing several skills. In theory: graduate to next section iff master all of its skills In practice: Student exhausts all available problems, gets promoted Teacher moves student to other section Student stops CTAI work altogether Final Reassigned Promoted Mastered 81% 95% Do students who are more likely to master worked sections have bigger treatment effects? Adam Sales Latent Principal Strata 5 / 27

8 How Important is Mastery Learning? Data We ve Got Z i Treatment assignment (randomized at school level) Adam Sales Latent Principal Strata 6 / 27

9 How Important is Mastery Learning? Data We ve Got Z i Treatment assignment (randomized at school level) Y i Post-test score Adam Sales Latent Principal Strata 6 / 27

10 How Important is Mastery Learning? Data We ve Got Z i Treatment assignment (randomized at school level) Y i Post-test score m is (for treatment group) Did student i master section s Adam Sales Latent Principal Strata 6 / 27

11 How Important is Mastery Learning? Data We ve Got Z i Treatment assignment (randomized at school level) Y i Post-test score m is (for treatment group) Did student i master section s x i Covariates: pretest, race, sex, special ed Adam Sales Latent Principal Strata 6 / 27

12 How Important is Mastery Learning? Data We ve Got Z i Treatment assignment (randomized at school level) Y i Post-test score m is (for treatment group) Did student i master section s x i Covariates: pretest, race, sex, special ed n = 5308 students. n T = 2390, n C = 2918 Adam Sales Latent Principal Strata 6 / 27

13 How Important is Mastery Learning? Data We ve Got Z i Treatment assignment (randomized at school level) Y i Post-test score m is (for treatment group) Did student i master section s x i Covariates: pretest, race, sex, special ed n = 5308 students. n T = 2390, n C = schools Adam Sales Latent Principal Strata 6 / 27

14 How Important is Mastery Learning? Data We ve Got Z i Treatment assignment (randomized at school level) Y i Post-test score m is (for treatment group) Did student i master section s x i Covariates: pretest, race, sex, special ed n = 5308 students. n T = 2390, n C = schools 86,677 worked sections Adam Sales Latent Principal Strata 6 / 27

15 Principal Stratification: Potential Mastery First Shot: m Take m i : the proportion of worked sections i mastered What s the average treatment effect for someone with m = m? Adam Sales Latent Principal Strata 7 / 27

16 Principal Stratification: Potential Mastery First Shot: m Take m i : the proportion of worked sections i mastered What s the average treatment effect for someone with m = m? Adam Sales Latent Principal Strata 7 / 27

17 Principal Stratification: Potential Mastery First Shot: m Take m i : the proportion of worked sections i mastered What s the average treatment effect for someone with m = m? Problem: that doesn t make any sense! Why not? Adam Sales Latent Principal Strata 7 / 27

18 Principal Stratification: Potential Mastery The Problem with Intermediate Variables Potential outcomes: Y Ti, Y Ci outcome if i is assigned to treatment (Z i = 1) / control (Z i = 0) Treatment effect: Y T Y C Average treatment effect (ATE): EY T EY C Adam Sales Latent Principal Strata 8 / 27

19 Principal Stratification: Potential Mastery The Problem with Intermediate Variables Potential outcomes: Y Ti, Y Ci outcome if i is assigned to treatment (Z i = 1) / control (Z i = 0) Treatment effect: Y T Y C Average treatment effect (ATE): EY T EY C ATE if m = m: E[Y T m = m] E[Y C m = m] But control group had no access to CTAI m is undefined if Z = 0! E[Y C m = m] makes no sense! Adam Sales Latent Principal Strata 8 / 27

20 Principal Stratification: Potential Mastery The Principal Stratification Solution Frangakis, C. E., & Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics, 58(1), Solution: consider m Ti What proportion of worked sections would i master if i is assigned to treatment A potential value E[Y C m T = m] For the subset of the population that would master m proportion of worked sections if assigned to treatment How would they score on the posttest if they are assigned to control? m T is a covariate Adam Sales Latent Principal Strata 9 / 27

21 Principal Stratification: Potential Mastery The Principal Stratification Solution Frangakis, C. E., & Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics, 58(1), Solution: consider m Ti What proportion of worked sections would i master if i is assigned to treatment A potential value E[Y C m T = m] For the subset of the population that would master m proportion of worked sections if assigned to treatment How would they score on the posttest if they are assigned to control? m T is a covariate... albeit only observed for the treatment group Adam Sales Latent Principal Strata 9 / 27

22 Principal Stratification: Potential Mastery The Principal Stratification Solution Frangakis, C. E., & Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics, 58(1), Solution: consider m Ti What proportion of worked sections would i master if i is assigned to treatment A potential value E[Y C m T = m] For the subset of the population that would master m proportion of worked sections if assigned to treatment How would they score on the posttest if they are assigned to control? m T is a covariate... albeit only observed for the treatment group Treatment effect E[Y T m = m] E[Y C m = m] principal effect Adam Sales Latent Principal Strata 9 / 27

23 Principal Stratification: Potential Mastery This is a Bit Different from Typical Principal Stratification m is continuous There s no such thing as m C Adam Sales Latent Principal Strata 10 / 27

24 Principal Stratification: Potential Mastery Model Based Estimation We don t know m T for the control group But we can identify its distribution f ( m T x) from the treatment group data Randomization guarantees f ( m T x, Z) = f ( m T x) i.e. we can extrapolate from the treatment group the the control group. Use MCMC to simultaneously fit models: Predict mt as a function of x Y as a function of Z, x, mt (Talk to Avi Feller et al. about how this doesn t work) Adam Sales Latent Principal Strata 11 / 27

25 Principal Stratification: Potential Mastery There Was An Attempt Linear (Normal) model for m T given x Linear (Normal) model for Y given Z, m T, and x Control Treatment Posttest Score m T Adam Sales Latent Principal Strata 12 / 27

26 Principal Stratification: Potential Mastery There Was An Attempt Linear (Normal) model for m T given x Linear (Normal) model for Y given Z, m T, and x Control Treatment Posttest Score m T Misspecified but that s the least of its worries Adam Sales Latent Principal Strata 12 / 27

27 We Need a Measurement Model! Does m Just Reflect the Number of Sections Worked? 100 nsec 50 n sec i number of sections i worked. Variation in m 1/n sec m Adam Sales Latent Principal Strata 13 / 27

28 We Need a Measurement Model! Does m Just Reflect the Difficulty of Sections Worked? Avg. Sec. Difficulty m Adam Sales Latent Principal Strata 14 / 27

29 We Need a Measurement Model! Solution: Latent Mastery! Enter IRT: measure (latent) ability from tests but abstracted from particular test. Adam Sales Latent Principal Strata 15 / 27

30 We Need a Measurement Model! Solution: Latent Mastery! Enter IRT: measure (latent) ability from tests but abstracted from particular test. IRT Items on a test CTAI Mastery CTAI sections Adam Sales Latent Principal Strata 15 / 27

31 We Need a Measurement Model! Solution: Latent Mastery! Enter IRT: measure (latent) ability from tests but abstracted from particular test. IRT Items on a test Item correctness CTAI Mastery CTAI sections Section mastery Adam Sales Latent Principal Strata 15 / 27

32 We Need a Measurement Model! Solution: Latent Mastery! Enter IRT: measure (latent) ability from tests but abstracted from particular test. IRT Items on a test Item correctness Item difficulty to get right CTAI Mastery CTAI sections Section mastery Section difficulty to master Adam Sales Latent Principal Strata 15 / 27

33 We Need a Measurement Model! Solution: Latent Mastery! Enter IRT: measure (latent) ability from tests but abstracted from particular test. IRT Items on a test Item correctness Item difficulty to get right Student ability CTAI Mastery CTAI sections Section mastery Section difficulty to master Student mastery propensity Adam Sales Latent Principal Strata 15 / 27

34 We Need a Measurement Model! Rasch Model for Mastery Pr(m is = 1) = logit 1 ( η }{{} Ti b s ) }{{} student mastery section difficulty Adam Sales Latent Principal Strata 16 / 27

35 We Need a Measurement Model! Rasch Model for Mastery Pr(m is = 1) = logit 1 ( η }{{} Ti b s ) }{{} student mastery section difficulty η Ti is potential mastery how i would mastered sections if assigned to treatment Multilevel: model η Ti as function of covariates x Then extrapolate to control group Adam Sales Latent Principal Strata 16 / 27

36 We Need a Measurement Model! Rasch Model for Mastery Pr(m is = 1) = logit 1 ( η }{{} Ti b s ) }{{} student mastery section difficulty η Ti is potential mastery how i would mastered sections if assigned to treatment Multilevel: model η Ti as function of covariates x Then extrapolate to control group η Ti is not observed for anyone in any control group Adam Sales Latent Principal Strata 16 / 27

37 We Need a Measurement Model! Does it Work? Part I 100 nsec m η T Adam Sales Latent Principal Strata 17 / 27

38 We Need a Measurement Model! Does it Work? Part II Avg. Sec. Difficulty m η T Adam Sales Latent Principal Strata 18 / 27

39 We Need a Measurement Model! Complex Model Many parts: Model m is as a function of η Ti Adam Sales Latent Principal Strata 19 / 27

40 We Need a Measurement Model! Complex Model Many parts: Model m is as a function of η Ti Model η Ti as a function of x Adam Sales Latent Principal Strata 19 / 27

41 We Need a Measurement Model! Complex Model Many parts: Model m is as a function of η Ti Model η Ti as a function of x Model Treatment effect as a function of η Ti Adam Sales Latent Principal Strata 19 / 27

42 We Need a Measurement Model! Complex Model Many parts: Model m is as a function of η Ti Model η Ti as a function of x Model Treatment effect as a function of η Ti Model Y as a function of x, η T, Z, treatment effect Adam Sales Latent Principal Strata 19 / 27

43 Results/Model Checking The Effect Posterior Average Posterior Draws 0.4 ˆτ(ηT ) η T Posterior Probability slope is negative 0.88 Adam Sales Latent Principal Strata 20 / 27

44 Results/Model Checking In Terms of Potential Outcomes Y C Y T E[Y Z η T ] η T Adam Sales Latent Principal Strata 21 / 27

45 Results/Model Checking Model Checking (One of many strategies) 1 Delete control group 2 Copy treatment group 3 Call this copy control 4 Delete its mastery data Adam Sales Latent Principal Strata 22 / 27

46 Results/Model Checking Model Checking (One of many strategies) 1 Delete control group 2 Copy treatment group 3 Call this copy control 4 Delete its mastery data This makes fake data: That reflects all the weirdness of the real data (at least the treatment group) With exactly no treatment effect We can simulate, add on, fake treatment effect Adam Sales Latent Principal Strata 22 / 27

47 Results/Model Checking Model Checking Results Posterior Average Posterior Draws True Effect 0.50 τ = 0 ˆτ = ηT τ = ɛ ˆτ = ηT ˆτ(ηT ) 0.50 τ = ηT ˆτ = ηT τ = 0.04x x ˆτ = ηT η T Adam Sales Latent Principal Strata 23 / 27

48 Discussion What Does this Mean for Mastery Learning? Not conclusive Mastery learning not the only thing at play Maybe it doesn t work at all On the other hand... 89% of students mastered most of the sections they worked Maybe η T is confounded with other stuff (it is) Adam Sales Latent Principal Strata 24 / 27

49 Discussion This is Good News Challenges one of the theories at the foundation of CTAI. But in practice: When η T is low, CTAI isn t working as intended But it still works! CTAI (appears to) tighten the gap between stronger and weaker students. Adam Sales Latent Principal Strata 25 / 27

50 Discussion What About Statistics? Why I m worried: Do latent potential values make sense? (I think so) What if the model s wrong?????????????????????? Adam Sales Latent Principal Strata 26 / 27

51 Discussion What About Statistics? Why I m worried: Do latent potential values make sense? (I think so) What if the model s wrong?????????????????????? Why I m excited Opens the door to principal stratification on more subtle/interesting intermediate variables Causal modeling of complex, high-dimensional, multilevel data For instance: cluster analysis! etc etc Adam Sales Latent Principal Strata 26 / 27

52 Discussion Thank You! Paper on arxiv: Code on github: Adam Sales Latent Principal Strata 27 / 27