Combining Scaling and Classification: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions

Transcription

1 Combining Scaling and Classification: A Psychomtric Modl for Scaling Ability and Diagnosing Misconcptions Lain Bradshaw Th Univrsity of Gorgia Cognition and Assssmnt SIG Businss Mting April

2 Ovrviw Sampl Multipl Choic Itm Ways to psychomtrically modl this itm Scaling Individuals and Classifying Misconcptions SICM modl Nw psychomtric modl dvlopd through my dissrtation Brif rsults from mpirical data analysis Forc Concpts Invntory

3 Exampl Multipl Choic Itm Which of th following oprations corrctly shows how to find th ara in inchs of a rctangl that is 3 ft long and 8 inchs wid? a 36 in. x 8 in. Corrct! b 8 in. x ¼ in. c 36 in in. + 8 in. + 8 in. d ¼ in. + ¼ in. + 8 in. +8 in. Confusion with convrting units Confuss ara with primtr Confusion with convrting units and ara Not uncommon to find itms writtn lik this on Incorrct options align to common studnt concptions or rrors In scinc.g. Hstns t al. 1992; Sadlr 1998; Sadlr t al In statistics.g. Garfild 1998; Khazanov 2009 In gnral tst dsign practics sk plausibl distractors How do w statistically capitaliz on this rich information?

4 Psychomtric Choics Th psychomtric modl chosn should Rflct mpirical thoris of th domain-spcific scinc Provid typs of information that tachrs and studnts sk Common choics Practical modl: CTT total scors and subscors Rsarch sttings: NR IRT modl Bock 1972 Captur th uniqu information in th itm rspons Itm rspons is a function of an ovrall continuous math ability Altrnat choic Scaling Individuals and Classifying Misconcptions SICM modl was tailord for this kind of itm Itm rspons is a function of: An ovrall continuous math ability Two catgorical misconcptions : 1 = confuss ara with primtr 2 = difficulty with multiplicativ comparisons ndd to mak convrsions among units Four possibl misconcption pattrns

5 NR IRT Itm Rspons Function IRT mthods scal xamins by locating thm along a singl continuum according to an ovrall ability Itm rspons is a function of studnt s ovrall continuous math ability A B C D

6 NR IRT Itm Rspons Itm rspons is a function of ability only Probabilitis This information is usful for: Comparing studnts abilitis for scholarship or awards Tracking growth on an assumd intrval lvl What about th rrors or misconcptions? Math Ability

7 What about th rrors or misconcptions? Frquntly CTT mthods ar usd Subscors for th numbr of tims a studnt slcts an incorrct altrnativ alignd to a misconcption Problms Small numbr of itms pr misconcption Ar itm rsponss indpndnt conditional on ability alon?

8 What about th rrors or misconcptions? Th SICM modl offrs an altrnat solution Harnsss practicality of diagnostic classification modls Provid mor rliabl multidimnsional fdback with small numbr of itms How? Us catgorical latnt variabls Includs misconcptions as a part of th itm rspons function Modls misconcptions as a latnt variabl

9 SICM Modl Itm Rspons Function In th SICM Modl th itm rspons is a function of 1. Ability as in th NR IRT modl Continuous trait 2. Attributs as in diagnostic classification modls DCMs Catgorical traits Attributs ar dfind as misconcptions instad of skills or abilitis SICM modl spcifications: A B C 1 2 D

10 SICM Modl Itm SICM modl stimats can b usd for: Classifying xamins according to misconcptions to tailor instruction or rmdiation Comparing xamins abilitis for ranking or accountability purposs

11 SICM Modl Itm Rspons Function for Exampl Itm D A B C 1 2 log b i a i b i i i A X P B X P λ λ λ + = = = log c i a i c i i i A X P C X P λ λ λ + = = = log d i d i d i a i d i i i A X P D X P λ λ λ λ λ = = = intrcpt loading for thta onto corrct altrnativ intraction btwn two misconcptions in D main ffcts for rspctiv misconcptions prsnt in incorrct altrnativs Modl is idntifid by stting th paramtrs in th baslin catgory A to zro and by standardizing th continuous prdictor.

12 Exampl Data Analysis

13 Forc Concpts Invntory FCI Tst that sks to idntify misconcptions studnts hav about Nwtonian forc concpts Carful tst construction fforts to writ incorrct options to b rflctiv of studnt misconcptions On of th most widly administrd tsts in physics ducation Purpos: illustrat SICM modl s us to scal ability and provid catgorical misconcption fdback

14 30 itm tst Forc Concpts Invntory FCI First 3 misconcptions masurd by at last 5 itms wr includd in th SICM modl Misconcption 1: imptus dissipation Misconcption 2: gradual/dlayd imptus build-up Misconcption 3: only activ agnts xrt forc Each misconcption masurd by 6 itms Masurd by 10 7 and 6 options rspctivly Data: high school studnts nrolld in a physics class

15 Rsults: Som Highlights Rlativ modl-data fit SICM modl was a mor parsimonious modl that th NR-IRT modl Diagnostic quality of incorrct options For avrag ability xamins possssing a misconcption incrasd th probability of slcting an alignd incorrct option on avrag by 10.8% 10.5% and 29% for Misconcptions 1-3 rspctivly

16 Rsults: Som Highlights Misconcption prvalnc % 33% % Imptus Dissipation Dlayd/Gradual Build-Up Activ agnts

17 Rsults for Studnts J and K 1 Itm J K CTT 2 3 SICM Misconcption No Misconcption

18 Rsults for Studnts J and K Itm J K CTT 2 3 SICM Itm J K CTT 0 4 SICM Itm J K CTT 3 1 SICM J K Total Scor 8 7 SICM SICM [001] [010] Misconcption No Misconcption Diagnosis J K Cut-off 2 [101] [110] Cut-off 3 [001] [110] Probabilistic [001] [010]

19 Simulation Study

20 Simulation Study Dsign Charactristics Valu or Intrval Tst Numbr of Itms Sampl Numbr of Examins 3000; Masurmnt Low High Modl Sampling intrval for intrcpts Structural Modl Sampling intrval for main ffcts Sampling intrval for λλ Sampling intrval for two-way intraction ffcts Highr-ordr intractions 0 Numbr of Attributs Misconcptions 36 Ttrachoric Corrlation among Attributs Distribution of Continuous Trait N01 64 Conditions 50 Rplications for Each Condition

21 Concluding Rmarks

22 Concluding Rmarks SICM modl addrsss a growing dmand for assssmnt systms: to gain mor fdback about what studnts do not undrstand Ranking individuals and providing diagnostic fdback ar two commonly co-occurring purposs of a tst that may b viwd as fundamntally antithtical purposs in commonly usd tsting paradigms Wainr t al pg. 342 SICM modl diagnostic fdback complmnts traditional masurs of ovrall ability Rliably masur and statistically account for misconcptions

23 Qustions? Commnts? Thank you!