On Comparability of Test Results from Computer Based and P&P Test Forms for SAM (Student Achievement Monitoring Toolkit)

Transcription

1 39 th Annual Conference IAEA 2013 International Assiciation of Educational assessment October 2013 Tel Aviv, Israel On Comparability of Test Results from Computer Based and P&P Test Forms for SAM (Student Achievement Monitoring Toolkit) Elena Kardanova Olga Maslenkova Center for International Cooperation in Education Development National Research University Higher School of Economics Russian Federation

2 Functional competence SAM SAM purpose: assessment of subject competences of primary school students in mathematics and native language. Theoretical framework : teaching/learning process concept based on L.S. Vygotsky s ideas Deep understanding and conceptual flexibility Conceptual understanding Action with comprehension Procedural knowledge Orientation towards external features of the problem The Zone of Proximal Development (ZPD) Curriculum learnt

3 SAM test structure Block 1 Item of the 1 st level (formal) Item of the 2 nd level (reflexive) Item of the 3 rd level (functional) Subtest 1 Each block includes three test items assigned to levels 1, 2, and 3 Test Block 2 Block n Item of the 1 st level (formal) Item of the 2 nd level (reflexive) Item of the 3 rd level (functional) Item of the 1 st level (formal) Item of the 2 nd level (reflexive) Item of the 3 rd level (functional) Subtest 2 Subtest 3 The test can be viewed as consisting of three subtests, each containing a set of problems at a given level of mastery across different content areas

4 SAM quality verification (P&P form) Sample: about 6000 fourth-grade students from different regions of the Russian Federation Modern test theory IRT was used as a basis for SAM assessment design A dichotomous Rasch model was selected as a model for test data modeling and students scaling Tests can be considered as essentially unidimensional All items demonstrate satisfactory psychometric characteristics and fit the model Validity study SAM tests can be acknowledged as a qualitative and valid measurement tool.

5 Estimation of examinees Mathematical competence scale Proficiency level Items of the 3 rd level Items of the 2 nd level Items of the 1 st level Student A Proficiency level 2 Student B Proficiency level 1 Student C Below level 1 Student D We expect student A to successfully complete at least 50% of level 3 items We expect student B to successfully complete at least 50% of level 2 items We expect student C to successfully complete at least 50% of level 1 items We expect student D to be unable to successfully complete even 50% of level 1 items

6 Two forms of SAM test administration: P&P and computer-based Advantages of CB form better secrecy of data reduction of cheating transmission of data in online mode auto-upload of data to database or information system reduction of time and financial expenses for test conduction and support autoscoring and immediate feedbacks on results automatic reports raise of interest for test from children diversity of items (animation, video, sounds, graphics, etc) Risk factors the level of computer skills of a student inability to assess the level of difficulty of the whole test and skipped parts Computer-based sometimes can worsen the perception of test item (for instance, a part of a text might not fit the screen) In computer-based form of SAM the items are presented in blocks, but one by one on a screen. Inability to see the whole block can also have an influence to the results of testing.

7 Description of computer version of SAM tets Computer based form of SAM was developed as a digital copy of P&P version The difference between P&P and computer-based forms is that student is not able to display the whole block, he can only view one item, not three. 2 algorithms used in computer-based version of SAM: Linear: Blocks of items are presented according the same sequence as they presented in P&P form. Nonlinear: Blocks of items are shuffled and the sequence of presenting to the student is not determined and individual.

8 Comparison of P&P and CB versions of SAM tests. Research questions Can we count 2 forms of compute-based testing (linear and randomized) as comparable and the results as not dependable on type of computerbased testing? Can we count P&P form and computer-based form as comparable and results as not dependable on type of testing? Could there be a common scale for presentation of results in all forms?

9 Comparison of P&P and CB versions of SAM tests. The method A common person study design Item level comparison Test level comparison Questionnaire

10 Experiment design The whole sample: 400 students 1 st stage. CB test with linear algorithm 100 students 1 st stage. CB test with non-linear algorithm 100 students 1 st stage. P&P test 200 students 2 nd stage. P&P test 200 students 2 nd stage. CB test with linear algorithm 100 students 2 nd stage. CB test with non-linear algorithm 100 students

11 Comparison of different forms of SAM: first results Stage Form Type of computer test Reliability 1 P&P 0,87 2 CB Linear 0,89 2 CB Randomized 0,86 Linear and randomized forms of CB testing can be recognized as equivalent The samples of students that took CB linear and CB randomized testing can be combined in terms of one stage.

12 Comparison of item difficulty in the two test formats IP IIK All items Mean Diff. (SD) 1 st level items Mean Diff. (SD) 2 nd level items Mean Diff. (SD) 3 rd level items Mean Diff. (SD) 1 Этап Бланковое 0,53 (0,28) 0,85 (0,07) 0,52 (0,13) 0,23 (0,1) 2 Этап Компьютерное 0,49 (0,25) 0,75 (0,17) 0,49 (0,10) 0,22 (0,1)

13 Conclusions: The majority of items demonstrate similar characteristics. Form of testing does not affect the difficulty of the items (student sees only one item in CB form and the whole block in P&P form). The correlation between the difficulty of the items in two forms equal to There are a few items in the test which difficulty is significantly different in CB and P&P forms. These are 19 (of the 1 st level), 23 (of 2 nd level), 25 (of 1 st level), 26 of (2 nd level). Wherein 3 items are easier in P&P form and 1 in CB form. For other items (which are less different) we can t say for sure which form is easier. Items of 3 rd level have close characteristics. Given that they are solved by only the best students, it can be assumed that the format does not affect the results of the best students. There is no evidence to suggest that by the end of the test, students are more tired, carrying out the test in the CB form.

14 Analysis of the item with the biggest difference in difficulty

15 DIF-analysis: special design

16 Two forms after deleting 4 items SUMMARY OF 162 MEASURED Student: P&P Version TOTAL MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.46 TRUE SD 1.16 SEPARATION 2.53 Studen RELIABILITY.87 MODEL RMSE.44 TRUE SD 1.16 SEPARATION 2.66 Studen RELIABILITY.88 S.E. OF Student MEAN = Student RAW SCORE-TO-MEASURE CORRELATION = 1.00 CRONBACH ALPHA (KR-20) Student RAW SCORE "TEST" RELIABILITY =.87 SUMMARY OF 162 MEASURED Student: CB version TOTAL MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.45 TRUE SD 1.10 SEPARATION 2.48 Studen RELIABILITY.86 MODEL RMSE.43 TRUE SD 1.11 SEPARATION 2.58 Studen RELIABILITY.87 S.E. OF Student MEAN = Student RAW SCORE-TO-MEASURE CORRELATION = 1.00 CRONBACH ALPHA (KR-20) Student RAW SCORE "TEST" RELIABILITY =.87

17 Comparison of TIF (after deleting 4 items) Red line: P&P test Black line: CB test

18 Comparison of students results % 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% Below 1st proficiency level 1st proficiency level 2nd proficiency level 3rd proficiency level P&P KB Below 1 st proficiency level 1 st proficiency level 2 nd proficiency level 3 rd proficiency level P&P 3.7% 38.3% 51.2% 6.8% KB 11.1% 38.3% 45.7% 4.9%

19 Distribution of examinees who kept or changed the proficiency level

20 What could affect the differences in test results in two forms? The hypothesis: Presence or absence of experience in using a computer while out of school affects passing the CB form of the test The difference in the processing of the test affects the time a child spends on the performance of items The reduction of speed of answering makes negative affect on results of testing. The need to understand the features of interface of CB form of test makes a negative affect on the test results

21 24.6% Questionnaire analysis How often do you use a computer at home? 3.2% 25.0% I have no a computer Less than one hour in a day 1-2 hours in a day Did you use a paper draft at computer test? 2.8% 44.4% Often Seldom Never 52.8% More than two hours in a day Did you understand how to transfer from one item to another one, come back, etc. at computer test? 85.2% 47.2% 14.1% I didn't try to understand I didn't understand It took time to understand It was clear for me Which text is more suitable for you to read? 22.2% 43.3% 34.5% Paper text The same Computer text

22 Analysis of questionnaire data Correlations CB test results P&P test results Question 1 How often do you use computer at home? Question 2 Did you use a paper draft at a computer test? Question 3 Did you understand how to transfer from one item to another one, etc Question 4 Which text is more suitable for you to read? Regression analysis Impact of computer skills is insignificant for CB test results Impact of technological features of CB test is insignificant

23 Conclusions The majority of items demonstrate similar characteristics in P&P and CB tests. Four items are functioning differently. Test characteristics after deleting 4 items are very similar The format does not affect the results of the best students Students distributions by proficiency levels from two forms are similar Impact of computer skills is insignificant for CB test results Impact of technological features of CB test is insignificant Thus, are P&P and CB SAM test forms equivalent? Research is ongoing

24 Thank you for your attention Center for International Cooperation in Education Development National Research University Higher School of Economics