Comparing Apples with Oranges? Linking National and International Large-scale Assessments

Transcription

1 Comparing Apples with Oranges? Linking National and International Large-scale Assessments Olaf Köller Leibniz Institute für Science and Mathematics Education (Kiel) and Centre for International Student Assessment (Munich) "Standard setting: International state of research and practices in the nordic countries Oslo, September 23, 2015 Prof. Dr. Olaf Köller, Leibniz Institute for Science and Mathematics Education

2 Comparing Apples with Oranges? Co-operation Project Annika Nissen Timo Ehmke Olaf Köller Christoph Duchardt Core Reference: Nissen, A., Ehmke, T., Köller, O. & Duchardt, C. (2015). Comparing apples with oranges? An approach to link TIMSS and the National Educational Panel Study in Germany via equipercentile and IRT methods. Studies in Educational Evaluation, 47, DOI /j.stueduc X. Prof. Dr. Olaf Köller, Leibniz Institute for Science and Mathematics Education 2

3 Starting point: Large-scale Assessments in Germany PIRLS u u TIMSS u u PISA u u u NA-PS u u NA-SS u u u NEPS u u u PIRLS: Progress in Reading Literacy Study TIMSS: Trends in Mathematics and Science Study NA- PS: National Assessment in Primary School (Grade 4; German, Mathematics) NA- SS: National Assessment in Secondary School (Languages vs. Math & Science) NEPS: National Educational Panel Study; Data Collections in Grades 5 and 9 Prof. Dr. Olaf Köller, Leibniz Institute for Science and Mathematics Education 3

4 Samples PIRLS: Nationally representative sample of approx. n = students at the end of grade 4 TIMSS: Nationally representative sample of approx. n = students at the end of grade 4 PISA: Nationally representative sample of approx. n = year old students and of approx. n = th graders NA-PS: Representative samples of all 16 federal states (approx. n = per state) at the end of grade 4 NA-SekS: Representative samples of all 16 federal states (approx. n = per state) at the end of grade 9 (approx. n = ) NEPS: Nationally representative samples of students at the beginning of grade 5 (n = 7.500) and at the beginning of grade 9 (n = ) Leibniz Institute for Science and Mathematics Education, Kiel, Germany

5 Test Designs PIRLS: Multi-matrix design; 80 minutes testing (3PL) TIMSS: Multi-matrix design; 80 minutes testing math and science (3PL) PISA: Multi-matrix design; 120 minutes testing; major domain and minor domains (1PL) NA-PS: Multi-matrix design; 80 minutes testing, parts of the sample only take mother tongue, others math, others mother tongue plus math (1PL) NA-SS: Multi-matrix design; 120 minutes testing, 60 minutes for each domain (1PL) NEPS: All students work on same items, 30 minutes math, 30 minutes science, 30 minutes reading (1PL) Leibniz Institute for Science and Mathematics Education, Kiel, Germany

6 Reasearch Questions Strong political as well as scientific pressure to link studies (see e.g., similar activities in USA, where NAEP 8 and TIMSS have been linked in 2011) Different tests, different constructs? Different tests, different proficiency level models? Can we use national tests to assess our students on international scales and vice versa? Leibniz Institute for Science and Mathematics Education, Kiel, Germany

7 Research Question I: Graphical Illustration Leibniz Institute for Science and Mathematics Education, Kiel, Germany

8 Reasearch Questions II and III: Graphical Illustration National Proficiency Levels International Proficiency Levels Level 5 Level 4 National Student and Item Sample Level 5 Level 4 Level 3 Level 3 Level 2 Level 1 International Student and Item Sample Level 2 Level Leibniz Institute for Science and Mathematics Education, Kiel, Germany

9 Linking TIMSS, PIRLS, National Assessment, and National Educational National Assessment German/Math 4 th graders (1 class per school) schools TIMSS/PIRLS 4 th graders (1 class per school) 201 schools TIMSS/NEPS/ National Assessment 4 th graders (1 class per school) 80 schools

10 Linking TIMSS, PIRLS, National Assessment, and National Educational TIMSS/PIRLS 4 th graders (1 class per school) 201 schools TIMSS/NEPS/ National Assessment 4 th graders (1 class per school) 80 schools

11 Both studies measure mathematics competencies Aim: Link these studies to use international benchmarks Often two linking-methods are distinguished: Classical Test Theory Equating Which method fits? Item-Response-Theory Equating the data better Leibniz Institute for Science and Mathematics Education, Kiel, Germany

12 Which linking method (CTT or IRT) should be preferred regarding the: (1) descriptive measurements? (2) the classification accuracy to the TIMSS International Benchmarks? (3) analysis of different subgroups? Leibniz Institute for Science and Mathematics Education, Kiel, Germany

13 Different conceptual approaches, but both with the aim to measure mathematics competencies at the end of primary school (TIMSS) respectively at the beginning of grade 5 (NEPS) Leibniz Institute for Science and Mathematics Education, Kiel, Germany

14 % Linking TIMSS and National Educational

15 78 Primary Schools in Germany 80 classes N = 733 fourth graders (52 % male, 48 % female) Leibniz Institute for Science and Mathematics Education, Kiel, Germany

16 Leibniz Institute for Science and Mathematics Education, Kiel, Germany

17 TIMSS 2011, Mathematics, Grade 4: 3-parameter IRT model Fixed item parameters from international database Transformation of Students PVs into international TIMSS achievement scale metric NEPS 2010, Mathematics, Grade 5: 1-parameter Rasch Model Fixed item parameters taken from NEPS 2010 Students WLEs transformed into only positive integer Leibniz Institute for Science and Mathematics Education, Kiel, Germany

18 Model 1 (2 Dim.) Model 2 (1 Dim.) N Parameter Deviance AIC BIC CAIC NEPS- LV 2- dim NEPS- LV 1- dim PL model;; Findings from ConQuest 3.0 Leibniz Institute for Science and Mathematics Education, Kiel, Germany

19 I. Classical Test TheoryEquating e.g. Equipercentile Equating: (Cartwright, 2012;; Hambleton et al., 2009) 1) Determine percentile ranks for the score distributions 2) Declare the scores with the same percentile as equivalent à Linking basis: Score distributions II. IRT Linking (Pietsch et al., 2009;; NCES, 2013) 1) Estimating item parameters 2) Scaling estimated parameters to a base IRT scale (linear transformation) 3) Transform true scores of new test form to true score scale on old form. à Linking basis: modeling student s responses to items Leibniz Institute for Science and Mathematics Education, Kiel, Germany

20 I. Equipercentile Equating (1) Finding percentile rank for each score value of the TIMSS-test and NEPStest (2) Matching the scores by the corresponding percentile values using the Software LEGS 2.01 (Brennan, 2004) Leibniz Institute for Science and Mathematics Education, Kiel, Germany

21 Results Equipercentile Equating NEPS score distribution TIMSS score distribution x* freq cum freq f(x) F(x) P(x) y** freq cum freq g(y) G(y) Q(y)

22 Results Equipercentile Equating TIMSS scores NEPS scores

23 IRT Linking (1) Scaling TIMSS & NEPS data simultaneously in a single IRT model by fixing item parameters of TIMSS international scale (2) Calibrating NEPS data with item parameters of the common scaling (3) Converting student s score into score equivalents in the TIMSS international scale Leibniz Institute for Science and Mathematics Education, Kiel, Germany

24 Results Equating Descriptive statistics for NEPS and TIMSS MEAN SD SKEW KURT NEPS TIMSS Equipercentile Equating IRT- Equating Classification of students to TIMSS International Benchmarks in Mathematics TIMSS 2011 International Benchmarks Cohen's < low low intermediate high advanced Sum Kappa TIMSS 1.2% 13.1% 38.6% 37.7% 9.4% 100.0% Equipercentile Equating 0.7% 14.3% 39.6% 37.2% 8.2% 100.0% IRT Equating 1,6% 13.4% 39.6% 33.7% 11.7% 100.0% Leibniz Institute for Science and Mathematics Education, Kiel, Germany

25 Results Equating IRT Linking Equipercentile Equating Sum Sum Leibniz Institute for Science and Mathematics Education, Kiel, Germany

26 Discussion Application of linking NEPS and TIMSS: Criterium-based interpretation of the NEPS mathematics scores Basis for longitudinal studies on students that fail the lowest (or reach the highest) educational standards In the Validation Study a) both methods lead to à Same estimates of population means à Classification accuracy of proficiency levels is satisfying à Similar Skwenes and Kurtosis b) Equipercentile methods should be prefered regarding à estimation of standard deviations Leibniz Institute for Science and Mathematics Education, Kiel, Germany

27 Thank you very much for your attention! Contact: Prof. Dr. Olaf Köller, Leibniz Institute for Science and Mathematics Education 27