Standard setting in DET

Transcription

1 Standard setting in DET Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 1

2 The Diagnostic Educational Test Subjects: writing English, writing Dutch, Mathematics Students halfway secondary education of five different educational streams Diagnosing the knowledge aspects and sub-skills that are necessary for a performance on the required level For each attribute, determine whether student Below required level At required level Above required level Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 2

3 Purpose Within each subject For each educational level Determine borders between below, at and above the required level for each attribute In total 15 standard setting sessions Method: Variation Data-Driven Direct Consensus Method (3 DC)?? Attribute Below level At level Above level Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 3

4 Standard setting method 3 DC method: Data-Driven Direct Consensus Method (Feskens, Keuning, Van Til & Verheyen, 2014) Starting point: method used for the ESLC (Verhelst, 2012) Contains elements from other methods, such as Bookmark method Angoff method Direct Consensus method Characterization Criterion-referenced Assessment-centered Analytical Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 4

5 Information Use as much information as possible to arrive at well-founded standards 1. Content description: Educational objectives (National institute of curriculum development, 2012) 2. Assessment a) Student models (Board of examinations, 2014) b) Clusters of tasks (one cluster for each sub-attribute) 3. Experience of experts (teachers) 4. Empirical information a) Reality feedback b) Normative feedback c) Impact feedback Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 5

6 Standard setting session Prior to the session a short training Then during the session: View clusters of tasks Task: How many responses of this cluster a student should have correct if his/her ability is right on the border of below level and at level of aspect XX? Two review rounds for each of the two borders Round 1 Each expert gives a judgment Normative feedback Discussion Round 2 Each expert gives a final judgment Normative feedback (Discussion) Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 6

7 Standard setting program 1. Information by mail: Letter, description of educational objectives and of student model 2. Training session: a) DET in a nutshell b) The student and task models c) Standard setting procedure d) The use of the standard setting tool 3. For each subject one standard setting session for each educational stream (total 15) Moderator (assessment experts): Training session Observation of standard setting Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 7

8 View tasks Previewer facet Comment paper Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 8

9 Example standard setting form (reality feedback) Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 9

10 Tool for standard setting Supports entire standard setting session 1. Judgment Two borders and two rounds 2. Viewing tasks 3. Providing feedback a) Reality feedback b) Normative feedback c) Impact feedback Computerized standard setting using 3DC: Downloads: Technical manual & program Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 10

11 SCREEN SHOTS DEMO TOOL Sanneke Schouwstra and Patrick de Klein Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 11

12 Standard setting form Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 12

13 Normative feedback: evaluations Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 13

14 Normative feedback: form Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 14

15 Impact feedback Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 15

16 END DEMO TOOL Sanneke Schouwstra and Patrick de Klein Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 16

17 Reporting and finalizing standards Analyses: Calculate the standard for each main attribute Check standards Report to external assessment committee External assessment committee Overview results standard setting sessions of all educational streams and subjects Evaluate quality Evaluate impact Advise to Board of Examinations Apply standards and calibrate tasks Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 17

18 Calculation standard for main attribute Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 18

19 Comparison educational streams Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 19

20 Report to external committee Within subject for each educational stream 1. Overview of judgments a) Measures b) Graphical (in form) 2. Calculated standards for main attributes 3. Impact Comparison educational streams Verbal report Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 20

21