PISA 2003 MAIN STUDY DATA ENTRY MANUAL

Transcription

1 PISA 2003 MAIN STUDY DATA ENTRY MANUAL Project Consortium: Australian Council For Educational Research (ACER) Netherlands National Institute for Educational Measurement (CITO group) Educational Testing Service (ETS) National Institute for Educational Policy Research (NIER, Japan) Westat

2

3 TABLE OF CONTENTS 1 INTRODUCTION KEYQUEST OVERVIEW AND INSTALLATION KEYQUEST OVERVIEW KEYQUEST INSTALLATION Installing KQADMIN and KQDATA Installing KQEntry PISA 2003 MAIN STUDY DATABASE PRE-CONFIGURED INSTRUMENTS List of Schools List of Students Student Tracking Form Study Programme Instrument Test Booklets Student and School Questionnaires Reliability Coding Sheets Session Report ISCO Codes Occupations CODEBOOKS STUDENT SAMPLING MINIMUM INFORMATION REQUIREMENTS Study Programme Information Preparing Study Programme Information for Student Sampling Examples how to Enter Study Programme Information into Relevant KeyQuest Instruments IMPORTING THE DATA INTO THE LIST OF STUDENTS SAMPLING FUNCTIONALITY CHANGING STUDENT TRACKING FORM LABELS PRINTING AND UPDATING STUDENT TRACKING FORM DATABASE ADAPTATIONS GENERAL COMMENTS QUESTIONNAIRE ADAPTATIONS daptations Required by the Consortium National Options Naming Convention Variable Modifications Removing Items from the Questionnaire and Changing the Order of Items Saving the Instrument ENTERING DATA KQADMIN AND KQENTRY CODING VARIABLES The Structure of Student and School Identification Missing, Invalid, and Not Applicable Codes Date and Time Variables Names Marker Identification Variables Test Item Variables Background Variables in the Questionnaires Coding Occupations Entering the <Mark> Information CLEANING PROCEDURES VALIDATION CHECKS REQUIRED FROM THE COUNTRY

4 7.1.1 Report 1: List of Schools School Questionnaire Consistency Report 2: Duplicate Booklets Report 3: Tracking Form Booklet Consistency Report 4: Tracking Form Questionnaire Consistency Report 5: Multiple Records in Reliability Booklets Report 6: Multiple Records in Reliability Booklets and Test Booklets Report 7: Marker ID Consistency VALIDATION CHECKS PERFORMED BY THE INTERNATIONAL STUDY CENTRE National Adaptations in the Database Verification of the Student Tracking Form and the List of Schools Verification of the Reliability Data Cognitive Data File Verification of the Context Questionnaire Data Verification of Within-School Sampling DATA SUBMISSION REFERENCES APPENDIX 1 CODEBOOKS Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet Name of Instrument: Booklet UH Name of Instrument: EC_ICT Name of Instrument: School Questionnaire Name of Instrument: Student Questionnaire Name of Instrument: List of Schools Name of Instrument: Student Tracking Name of Instrument: Study Programme...4 1PPENDIX 2 ISO COUNTRY AND AREA CODES APPENDIX 3 ISCO OCCUPATIONAL CODES

5 TABLE OF FIGURES Figure 1: Mapping a network drive...4 Figure 2: KQAdmin folder on KeyQuest CD Figure 3: KQAdmin setup file selected... 4 Figure 4: KeyQuest Setup Program Welcome Screen... 4 Figure 5: KeyQuest Setup Program Change Folder Screen... 4 Figure 6: KeyQuest Setup Program Change Installation Drive... 4 Figure 7: KeyQuest Setup Program Installation Drive Changed... 4 Figure 8: KeyQuest Setup Program Create New Folder Prompt... 4 Figure 9: KeyQuest Setup Program Installation Folder Created... 4 Figure 10: KeyQuest Setup Program Install Button... 4 Figure 11: KeyQuest Setup Program Setup Completed... 4 Figure 12: Start Menu Launching KeyQuest... 4 Figure 13: KeyQuest Cannot Find KQdata... 4 Figure 14: KeyQuest Find KeyQuest Folder on Network... 4 Figure 15: KeyQuest KQdata file found... 4 Figure 16: KeyQuest KQAdmin Main Menu... 4 Figure 17: KQEntry Folder on KeyQuest CD... 4 Figure 18: KQEntry Installation Welcome Screen... 4 Figure 19: KQEntry Installation Specify Installation Folder Screen... 4 Figure 20: KQEntry Installation Install Button... 4 Figure 21: KQEntry Installation Complete... 4 Figure 22: Start Menu Launching KQEntry... 4 Figure 23: Locate KQAdmin File... 4 Figure 24: KQAdmin Main Screen Student Sampling... 4 Figure 25: Student Sampling Screen... 4 Figure 26: Book Allocation Screen...4 Figure 27: Warning message... 4 Figure 28: Student Tracking Form Screen... 4 Figure 29: Tracking Form Labels Screen... 4 Figure 30: Edit / Create Variable Multiple choice Item as Vector Variable... 4 Figure 31: Edit / Create Variable Item With Length 1 and Width Figure 32: KQentry Management Screen... 4 Figure 33: Screen For Selecting An Instrument... 4 Figure 34: Main Screen of KQAdmin... 4 Figure 35: The Data Entry Screen for Booklet 1 page Figure 36: Status bar for School Name Variable

6 TABLE OF TABLES Table 1 SPT for a country where all 15-year-old students attend the same study programme... 4 Table 2 SPT for a country with one study programme per school... 4 Table 3 SPT where study programmes contain more than one ISCED level... 4 Table 4 SPT for a country with different programmes within schools... 4 Table 5 List of Students in Excel format... 4 Table 6 Adaptation to the item that does not require adaptation to the variable... 4 Table 7 Adaptation to the MC item (original variable name)... 4 Table 8 Adaptation to the MC item (national variable name)... 4 Table 9 Example of adaptation to the Q16, language spoken at home... 4 Table 10 Example of adaptation to the Q15a, country of birth... 4 Table 1daptation to the multiple-response item created as a number of variables... 4 Table 12 Adaptation to the multiple-response item that requires additional national variables... 4 Table 13 Adaptation of the multiple-response item created as a vector... 4 Table 14 Adapting open format item created as a numeric variable... 4 Table 15 Removing items from the questionnaire and KeyQuest... 4 Table 16 Removing items from the questionnaire but not from KyQuest Table 17 PISA Test Item Types and Data Entry Requirements... 4 Table 19 Numeric <Mark>information in QAS... 4 Table 20 Categorical <Mark> information in QAS

7 PISA 2003 MAIN STUDY DATA ENTRY MANUAL 1 INTRODUCTION 1. This document outlines the data management procedures for the PISA 2003 Main Study. The data management software KeyQuest is used in PISA for this purpose. This document should be read in conjunction with the KeyQuest Manual, that contains a detailed description of all generic features of the software. 2. This document consists of seven sections. The first section contains an overview of the KeyQuest generic and PISA specific features and illustrates installation of KeyQuest. 3. The second section describes the KeyQuest database and all pre-configured for the PISA 2002 Main Study instruments. It also contains a description of codebooks for these instruments. Codebooks are provided in the Appendix The third section describes student sampling with KeyQuest, which is a requirement for the PISA 2003 Main Study. 5. The fourth section outlines how to make national adaptations to PISA Questionnaires and corresponding adaptations to KeyQuest. It also explains how to record these adaptations in the Questionnaire Adaptation Spreadsheet (QAS). 6. The fifth section is a detailed description of the data entry procedures. This section also contains a description of the various ways of entering occupational data. 7. The sixth section describes the data cleaning procedures that countries are required to carry out before they submit their data and the data cleaning procedures that ACER will perform while preparing national data for the international database. 8. Finally, the seventh section summarizes the procedures and timeline for submitting the data. 2 KEYQUEST OVERVIEW AND INSTALLATION 2.1 KEYQUEST OVERVIEW 9. KeyQuest is generic software for data entry and data management. It is also used for student sampling and school reports. Student sampling with KeyQuest, is a requirement for the PISA 2003 Main Study. The upgraded version of KeyQuest contains the database structures for all the booklets, questionnaires and tracking instruments used in the PISA 2003 Main Study. 10. It is strongly recommended that data be entered directly into KeyQuest, in order to take advantage of its many PISA-specific features. During data entry, KeyQuest continuously monitors for two kinds of data entry errors: violations and Primary Key violations. 11. s are rules associated with each variable for ensuring that only legitimate data are entered. The s are specified when the variables are created, and they are used to check the entered data. See the KeyQuest Manual Chapters 5 and 6 for more details. Validation rules for each item are provided in the international Codebooks. 12. Primary Keys are used to ensure that any student identification or school identification entered in one the KeyQuest instruments is consistent with a student identification listed in the Student Tracking Form, or with the school identification listed in the List of Schools. It also guarantees that no student or school identification is entered twice in the same instrument. 7.

8 PISA 2003 Main Study data MUST BE SUBMITTED to the International Processing Centre WITHIN KEYQUEST whatever system is used at the National Centre for data entry. 2.2 KEYQUEST INSTALLATION 13. Ask your network administrator to create a folder on a network. In the example below we named the folder keyquest, and it is in the drive named Pisa\PisaAdmin, however you can use any convenient names Installing KQADMIN and KQDATA NOTE: The following steps need to be done only ONCE on any machine on the network. 14. On any operator machine map the allocated network drive. This can be done as follows: Open Windows Explorer Go to the Tools menu Select Map Network Drive Locate the folder that the network administrator has given you in the Shared Directories box. 15. You should now see the name of the folder in Explorer in the list of Folders Figure 1: Mapping a network drive. 8.

9 16. Insert the KeyQuest CD into your CD drive. 17. To install KQAdmin and KQdata choose KQAdmin folder in KeyQuest CD. Choose this folder Figure 2: KQAdmin folder on KeyQuest CD. 9.

10 18. Double click on the setup file (setup.exe). 19. The welcome screen appears. Figure 3: KQAdmin setup file selected. 20. Click Continue. Figure 4: KeyQuest Setup Program Welcome Screen 10.

11 21. Click Change Folder Figure 5: KeyQuest Setup Program Change Folder Screen 11.

12 22. Change the Drive (at the bottom of the screen) to the drive that you ve just mapped (in our example this is the F drive in Figure 7). Type keyquest in the pass. Change this Figure 6: KeyQuest Setup Program Change Installation Drive to this. Figure 7: KeyQuest Setup Program Installation Drive Changed 12.

13 23. Click OK. 24. Click Yes to create KeyQuest folder. 25. Click OK. Figure 8: KeyQuest Setup Program Create New Folder Prompt The folder has changed to F:/KeyQuest Figure 9: KeyQuest Setup Program Installation Folder Created 13.

14 26. Click on the Install button (this is a large button with a computer icon). If the warning to close all applications appears and all applications are closed, ignore the warning. Install Figure 10: KeyQuest Setup Program Install Button 27. The setup procedure will add KeyQuest to your start menu. KeyQuest will notify you when the setup has been completed successfully. Click OK to exit setup. Figure 11: KeyQuest Setup Program Setup Completed 14.

15 28. From the Windows Start Menu, select Programs\KeyQuest\PISA2003. Figure 12: Start Menu Launching KeyQuest 29. Enter the password to log on as Administrator. You will be given the message that KeyQuest could not locate the database KQdata. Figure 13: KeyQuest Cannot Find KQdata 15.

16 30. Click OK. 31. You must locate KQdata.mdb for KeyQuest. Select the KeyQuest Folder. Find the KeyQuest folder on the network drive. Figure 14: KeyQuest Find KeyQuest Folder on Network 32. Double Click on KQdata. KeyQuest will start. Figure 15: KeyQuest KQdata file found 16.

17 33. Click the Exit Data Base button and KQdata and KQAdmin have successfully been installed on the Network. Figure 16: KeyQuest KQAdmin Main Menu NOTE: In KeyQuest, the data entry screen is stored in KQAdmin.mdb while the data is stored in KQdata.mdb. It is possible for KQAdmin.mdb and KQdata.mdb. to become incompatible particularly in a network environment if multiple installations of KQAdmin.mdb are used to edit or create instruments. To avoid this problem, we strongly recommend the use of ONE DESIGNATED INSTALLATION of KQAdmin.mdb for performing any function other than data entry, exporting or importing data. This refers to all adaptations of the KeyQuest instruments and also to the use of the Study Programme instrument 17.

18 2.2.2 Installing KQEntry NOTE: The following steps must be done on EVERY machine to be used for data entry (including the machine which you used to install KQAdmin). 34. On each of the data entry machines insert the KeyQuest CD. Go to KQEntry folder. 35. Double click on the setup file (setup.exe). KQEntry folder on CD Figure 17: KQEntry Folder on KeyQuest CD 36. A welcome screen will appear. Click continue. Figure 18: KQEntry Installation Welcome Screen 18.

19 37. Click OK. Figure 19: KQEntry Installation Specify Installation Folder Screen 38. Click Install. If the warning to close all applications appears and all applications are closed, ignore the warning. Figure 20: KQEntry Installation Install Button 19.

20 39. Click OK. Figure 21: KQEntry Installation Complete 40. The setup procedure will add KQEntry to your start menu. 41. From the Windows Start Menu choose - Programs\ PISA Main Study2003 Entry.\PISA Main Study2003 Entry. Figure 22: Start Menu Launching KQEntry 20.

21 42. Enter the password to log on as Administrator. You will be asked to find KQAdmin. 43. Select KeyQuest Folder, then KQAdmin. Figure 23: Locate KQAdmin File 44. On the Data Entry Screen click the Setting Instruments button. 45. Select instruments that will be entered from this computer by placing them from the left side of the window to right side. The instruments have to be moved one by one. 46. KQEntry is now ready to use 21.

22 3 PISA 2003 MAIN STUDY DATABASE 47. The upgraded version of KeyQuest contains the database structures for all the booklets, questionnaires and tracking instruments used in the PISA 2003 Main Study. The Codebooks of the instruments can be produced by KeyQuest. Appendix 1 contains the Codebooks of the international version of all instruments and second section of this part of the manual contains explanations on how to read them. If any structural changes are made to an instrument, it is recommended to produce the national version of the codebook for it. 3.1 PRE-CONFIGURED INSTRUMENTS 48. The PISA 2003 Main Study version of KeyQuest is pre-configured with all of the international instruments List of Schools 49. All participating schools must be entered into the List of Schools instrument. This instrument is REQUIRED when you draw the student samples with KeyQuest and for the study programme column of the Student Tracking Form. 50. The List of Schools will also be useful if you need to produce school reports. Although the school name is optional, KeyQuest will require this information when generating the school report. It is therefore recommended that the school name field be completed. 51. There are merely a few adaptations that can be made to the structure of the list of schools. Only the following variables can be deleted SCHCORD1"School Coordinator Title" SCHCORD2"School Coordinator Surname" SCHCORD3"School Coordinator Given Name" 52. National variables independent from the first nine variables, for example, "Test Administrator Surname" and the like can be added if needed. 53. NO CHANGES are permitted to the first nine variables of this instrument StIDStrt "Stratum ID" StIDSch "School ID" SchName "School Name" SCHPSTAT "School Participation Status" SCHREP "School Sample Status" SCH15SAM "15-year-old sample size" SCH15NUM "School Number of 15-year-olds" SCHLSNUM "School Number of Students for Sampling" TRC2Q07 "Tracking Study Programme" 54. Data Entry into this instrument should precede student sampling and Data Entry into all other KeyQuest instruments List of Students 55. The List of Students is used by the Student Sampling Functionality of KeyQuest. The information stored in this instrument is used when building data in the Student Tracking instrument. Details on student sampling are given in the Chapter 5 of the PISA 2003 Main Study NPM Manual and in the appendices of the KeyQuest Manual. 56. Data Entry into this instrument is optional. 22.

23 57. However if the List of Students is used, Data Entry into it should precede student sampling and NO CHANGES are permitted to the following variables of this instrument StIDStrt "Stratum ID" StIDSch "School ID" SchName "School Name" TRC1Q02 Tracking Line Number TRC1Q04 Tracking Grade TRC1Q05 Tracking Gender TRC1Q06M Tracking Month of Birth TRC1Q06Y Tracking Year of Birth TRC2Q07 Tracking Study Programme Student Tracking Form 58. The instrument is Student Tracking. Details about the main study Student Tracking Form (STF) are given in the Chapter 5 of the PISA 2003 Main Study NPM Manual. Explanations of how to use the new column (7), Study Programme, of the STF, are also given in the Study Programme part of the Entering Data section of this manual. 59. NO adaptations to the STRUCTURE of this instrument are permitted. Some adaptations to the TEXT are REQUIRED and is explained in the Study Programme part of this manual Study Programme Instrument 60. The information stored in the Study Programme is used when recoding data in the Student Tracking instrument to the international format. The use of this new instrument is explained in the Study Programme part of the Entering Data section of the current document. Chapter 5 of the PISA 2003 Main Study NPM Manual also contains information about this new instrument. 61. NO adaptations to this instrument are permitted. Data Entry into this instrument should precede student sampling Test Booklets 62. The instrument names of thirteen test booklets in KeyQuest are Booklet 1 to Booklet 13. Booklet UH for special education students is also available as an International Option. 63. NO adaptations to TEST ITEMS are permitted. 64. Some Marker ID variables can be deleted if the country uses the same markers for different domains. Students demographic variables also can be dropped Student and School Questionnaires 65. The instrument names are School Questionnaire, Student Questionnaire, Student Questionnaire_EC, Student Questionnaire_ICT, Student Questionnaire_EC_ICT, Student Questionnaire_ICT and EC_ICT. 66. Those countries that do not use any International Options should use School Questionnaire and Student Questionnaire. Those countries that use only one International Option should use School Questionnaire and Student Questionnaire_EC, or Student Questionnaire_ICT. Those countries that use both International Options and do not add any national variables to the Student Questionnaire or add very few should use School Questionnaire and Student Questionnaire_EC_ICT. However, those countries that use both International Options and add some national variables to the Student Questionnaire should use School Questionnaire and Student Questionnaire, and EC_ICT as a separate instrument, because of the limited capabilities of Microsoft Access. 23.

24 67. Adaptations to the questionnaire database MUST REFLECT approved adaptations of the Questionnaires themselves. Adaptations of the KeyQuest Questionnaire Instruments are explained in the Questionnaire Adaptations part of this section. We also explain there how to record these adaptations on the QAS Reliability Coding Sheets 68. The reliability design is described in Chapter 7 of the PISA 2003 Main Study NPM Manual. The KeyQuest instruments associated with each type of Reliability Coding Sheet are included in KeyQuest. These instruments are: Reliability Booklet 1 (Math), Reliability Booklet 2 (Math), Reliability Booklet 3 (Math), Reliability Booklet 4 (Math), Reliability Booklet 5 (Math), Reliability Booklet 6 (Math), Reliability Booklet 8 (Science), Reliability Booklet 10 (Reading), Reliability Booklet 12 (PS). 69. NO adaptations to these instruments are permitted Session Report 70. The Session Report is used to capture Questions 1 to 19 from the Session Report Form which is completed by the Test Administrator (see Session Report Form in Test Administrators Manual). 71. Question Number 2 on the Session Report Form refers to the "PISA Identification Number". This is made up of the Stratum ID followed by School ID (e.g ). 72. Question Number 3 on the Session Report Form refers to the "School Identification Number" and refers to any national identification number used within the country 73. KeyQuest requires a Session Number when entering data from the Session Report Form. The Session Number is a unique number for a particular session within a school. This unique number can be assigned by the National Centre. 74. NO adaptations to this instrument are permitted ISCO Codes 75. The ISCO CODES instrument is required to build the List Based Responses variables used in the student questionnaires for occupation data. Details about ISCO codes and List Based Responses are described in the Data Entry Manual and in the Coding Occupations part of the Entering Data section of this document Occupations 76. The Occupation instrument is provided for the purpose of keying occupational data. 3.2 CODEBOOKS 77. The codebooks for the international version of all instruments are available in Appendix 1 of this document. 78. The codebooks for the national versions of the questionnaires should be exported directly from KeyQuest after all adaptations are made. Refer to KeyQuest Manual on how to export data from KeyQuest. All deviations from the international versions should be described in the QAS. 79. Codebooks contain a range of information about the KeyQuest variables. The following examples show two items from the Study Programme Codebook. 24.

25 TRP1Q Programme STF code StudyProg Between 1 and 40 or 97 or 98 or 99 TRP1Q Programme orientation Orientation "1" or "2" or "3" Value Value Label 1 General 2 Pre-Vocational 3 Vocational 80. The information in these examples should be read as follows: The variable names are TRP1Q02 and TRP1Q09, and these are the names by which these variables can be found in KeyQuest. Text 3 15 and indicates the position of the variables in the database in the Text Format, e.g. TRP1Q02 is located in columns 3-15 and TRP1Q09 is located in column 94. The item names are Programme STF code and Programme orientation. The KQEntry labels, which appear on the data entry screen, are StudyProg and Orientation. The lists all values that can be entered for multiple choice variable TRP1Q09 or range of values for numeric variable TRP1Q02. For more information on validation rules see KeyQuest Manual. The Value Label provides a meaning for each of possible value of the multiple choice variable TRP1Q09. 4 STUDENT SAMPLING 81. The student sampling functionality of KeyQuest was especially developed for the PISA Project. It samples students from the List of Students or from the information contained in the List of Schools if no students are entered in the List of Students. It uses a systematic sampling procedure by computing a sampling interval, generates automatically the student tracking form and assigns one of the rotated booklets to the students. 4.1 MINIMUM INFORMATION REQUIREMENTS 82. The minimum information that KeyQuest requires to be able to draw a student sample for a school is a record in the List of Schools with the following variables keyed in it. 25.

26 KQEntry Label 1 Variable Name Comment Stratum ID StIDStrt School ID StIDSch Participation SCHPSTAT School participation status:ccepted; 2 Refused; 3 Pending Main/Repl SCHREP School sample status 0 Main; 1 Replacement 1; 2 Replacement 2 MOS15 SCHMOS15 School Measure of size for 15-year-olds, e. g. total enrolment of 15 year-olds as reported in school sample forms (usually previous year enrolment), Sample Size SCH15SAM 15-year-olds sample size, typically 35 LIST15 SCH15NUM School Number of 15-year-olds, as indicated in the Student List received from school. LISTTOT SCHLSNUM School number of students for sampling, in case of oversampling, for example, if grade sample is required. If over-sampling is not required by the country, SCHLSNUM= SCH15NUM StudyProg TRC2Q07 A relevant Study Programme Code as indicated in the second column of the study programme table, if the school has only one programme for 15-years old. N/A code if there is a possibility in the school for different programmes within it for 15-years old. See next section for more details on Study Programme. 83. The school name and all information concerning the school coordinator is optional. Nevertheless, it is advised to provide the school name because it will appear on the student tracking form and also on the school report. In any cases, the variables listed above should not be deleted or modified. Indeed, these variables are used by KeyQuest and the program will fail if the variable has been dropped from the instrument. 84. If student information is not entered in the instrument List of Students, samples can still be drawn using just the information in the school variables mentioned above. KeyQuest will draw the student sample and generate the Student Tracking Form (STF) with the following information: student identification, line number in the list of students the booklet number assigned to each sampled student and the programme of study, IF there is only one programme in the school and the relevant code was entered into the List of schools. 85. Other variables such as gender, date of birth or programme of study, can be entered into the Student Tracking instrument using the Data Entry functionality. 86. If the List of Students is imported or entered for the school, and the format of student background variables was not changed, then the sampling functionality of KeyQuest will insert the information contained in this instrument when building the STF, i.e. the student name, the grade, the gender, the date of birth, and the programme of study. 1 Note that KQEntry Label refers to the labels on the Data Entry Screen, and Variable Name represents variables in the KeyQuest database. These names are not shown on the screen but can be found in the codebook 26.

27 Please note that if the format of the grade, the gender, the date of birth, or the programme of study variables were changed, this information will not show on the STF 87. Before drawing the sample, KeyQuest will check the consistency of the student enrolment data of the List of Schools (SCHLSNUM and SCH15NUM) with the number of records of the List of Students. In case of inconsistency, a warning message will appear on the screen and it will not be possible to draw the sample until data are consistent from both sources. Please note: Line# variable on the List of schools should begin from for EACH school Study Programme Information 88. For the PISA 2003 Main Study information about the study programmes of students will be collected through the Student Tracking Form (STF), Student Questionnaire, and a new KeyQuest instrument, Study Programme. Changes have been made to the List of Schools, List of Students, and STF to allow a different way of collecting this information. This section outlines how this information should be gathered and entered into KeyQuest. 89. Based on the information collected during the Field Trial the consortium provided a draft Study Programme Table (SPT) with a list of national study programmes and their respective codes to be used in the STF together with information on first and last grade, as well as their mapping to ISCED level, designation and orientation. NPMs should review this list of programmes and discuss any need for modifications with the consortium. 90. Once the content of the SPT has been agreed upon between NPMs and ACER, the list of programmes is added to the School Coordinator s (SC) and test administrators (TA) Manuals by the NPM and is part of the adaptation process to the TA and SC Manuals. 91. Information from the SPT should be entered into the Study Programme instrument of KeyQuest even if there is only one programme for the country Preparing Study Programme Information for Student Sampling 92. The following steps should be performed prior to printing of the STF. 93. Enter information into the Study Programme instrument of KeyQuest (see Entering Data chapter of this Manual). This new instrument reflects Study Programme Table (SPT). It contains all variables of this table except the following three: National name of programme, English/French programme description, and Comments. Information to this instrument should preferably be entered prior to all other information and definitely before the Student Tracking Forms (STF) are printed. 94. Enter study programme information as well as all other compulsory information into the List of Schools. The next section illustrates how study programme information can be entered into the List of Schools. 95. Enter study programme information into the List of Students if required (see section ). 96. Change study programme names in the Student Tracking instrument according to the national study programme names in SPT (see section ) 27.

28 4.1.3 Examples how to Enter Study Programme Information into Relevant KeyQuest Instruments. 97. Some countries have only one study programme per school, others can have different programmes within a school. In the first case the programme code has to be entered in the List of Schools for each school, in the second case it is recommended to enter it in the List of Students for each student. 98. Table 1 shows how the Study Programme Table (SPT) would be completed for a country where all 15-year-old students attend the same study programme (for example, all students study ISCED 2A, general). 99. In this case the variable Programme Code in the List of Schools has to be entered as 1 for all schools and KeyQuest will automatically insert this code for each student into Column 7 of the STF when they are produced at the sampling stage Tables 2 and 3 shows a completed SPT for a country with one study programme per school (for example, where the study programme is defined by the school type). In Table 2 National name of the programme refers to a unique combination of school type and ISCED level, while in Table 3 the same National name of programme refers to different ISCED levels. However, these ISCED levels could be identified by the student s grade In both cases the variable Programme Code in the List of Schools has to be entered corresponding to the school type and KeyQuest will again automatically insert the study programme codes for each student into the STF when they are produced. These codes are different for different school types Table 4 shows how the SPT would be completed for a country with different study programmes within schools. In this case variable Programme Code in the List of Schools has to be entered as N/A (97). KeyQuest will NOT automatically insert the study programme codes into STF for each student UNLESS this information is entered in the List of Students In countries with different study programmes per school it is a task of the School Coordinator to provide information about students participation in different Study Programmes available in the school If this information is provided electronically, the Programme Code could be imported into List of Students instrument together with other information about all 15-year-olds provided by school and then it will be inserted by KeyQuest into the STF when it is produced at the sampling stage If information about individual students participation in different study programmes is not available at the sampling stage, then this information should be written down directly into Column 7 of the printed STF by School Coordinator. The Data Entry Operator (DEO) will then enter this information into variable TRC2Q07 (Programme) of the Student Tracking Instrument when the STF is returned by the TA. 28.

29 Table 1 SPT for a country where all 15-year-old students attend the same study programme Country: Zland NPM: <name> Date: <date> Programme No. Programme Code in STF National name of programme English/French programme description First Grade Last Grade ISCED level ISCED designation Orientation 1 1 <Programme 1> Low Secondary school A General Comments All 15-year-olds are in the same programme. Note: For some schools Grade is entered into STF as 1 5, which corresponds to 6-10 Table 2 SPT for a country with one study programme per school Country: Xland NPM: <name> Date: <date> Programme No. Programme Code in STF National name of programme English/French programme description First Grade Last Grade ISCED level 1 1 <Programme 1> Basic education A General ISCED designation Orientation Comments 2 2 <Programme 2> Technical secondary school Vocational 3 3 <Programme 3> Gymnasium A General Programme depends on school type. Information is entered into the School List 29.

30 Table 3 SPT where study programmes contain more than one ISCED level Country: Yland NPM: <name> Date: <date> Programme Programme National name of English/French programme First Last ISCED ISCED No. Code in STF programme description Grade Grade level designation Orientation Comments Secondary school and 1 1 <Programme 1> Secondary school (lower) A General college are different institutions 2 1 <Programme 1> Secondary school (upper) A General same study programme for upper secondary 3 2 <Programme 2> Technical college (lower) C Vocational 4 2 <Programme 2> Technical college (upper) Vocational same study programme for upper secondary Table 4 SPT for a country with different programmes within schools Country: Yland NPM: <name> Date: <date> Programme No. Programme Code in STF National name of programme English/French programme description First Grade Last Grade ISCED level 1 1 <Programme 1> General Secondary education 7 (lower) 9 2 A General 2 1 <Programme 1> General Secondary education 10 (upper) 12 3 A General 3 2 <Programme 2> Vocational studies (lower) C Vocational ISCED designation Orientation Comments 4 2 <Programme 2> Vocational studies (upper) Vocational same study programme for upper secondary same study programme for upper secondary 30.

31 4.2 IMPORTING THE DATA INTO THE LIST OF STUDENTS Detailed information on how to import data is provided in Chapter 5 of the KeyQuest Manual. If data are imported using an Excel format, then the first line of the spreadsheet should contain the variable names, as shown in Table 5. Table 5 List of Students in Excel format SchNam StIDSt StIDSc TRC1Q0 TRC1Q0 TRC1Q0 TRC1Q0 TRC1Q06 TRC1Q06 TRC2Q0 School Name School Name School Name School Name School Name School Name School Name School Name School Name School Name School Name School Name School Name School Name School Name School Name School Name School Name School Name Variable names refer respectively to the school name, stratum identification, school identification, line number, student name, grade, gender, month of birth, year of birth and study program. Variables TRC1Q04 and TRC1Q07 are numeric, defined in Excel in General format. All other variables are alphanumerical. Alphanumerical variables will be defined in text format in Excel files. Please note that if you are importing several schools simultaneously, line number (TRC1Q02) for each school should begin with When importing data, it is possible to append to existing data or to replace all of the existing data. As lists of students may be imported at different times, it is necessary to have an APPEND option To do this, click on the Import Data button of the KQAdmin Main Screen (see Figure24). Choose List of Students from the drop down list, choose Append to existing data option and then brose the file If, for any reason, a list of students must be re-imported for one school, the records in the list of students must be firstly deleted one by one using the Data Entry functionality of KeyQuest Another way to solve this problem is to export the instrument, delete the records for that school in Excel, for instance, and then to re-import the whole file using the Replace existing data option. 31

32 The replace option should be used with extreme care because it will delete all data contained in the list of students, not only the data of the school, which must be re-imported. 4.3 SAMPLING FUNCTIONALITY 112. Click on Student Sampling button on the main screen to access the Student Sampling Screen. Figure 24: KQAdmin Main Screen Student Sampling 113. Student Sampling Screen will appear (Figure 25) Before drawing student sample, check booklet allocation. The Book Allocation button on the Student Sampling Screen leads to a screen, which allows a range of booklets or a single booklet to be allocated to the students in the selected schools. 32

33 Figure 25: Student Sampling Screen 115. The Range of Booklets option is selected by default. The booklet numbers will range from 1 to For those countries that use Booklet UH, when sampling students from the special education schools the one booklet option should be selected. Figure 26: Book Allocation Screen 117. After the relevant option is chosen, click on Return to Previous button Double click on the school name in the left window for which a student sample is to be drawn (Figure 25). The school name will move to the right window. Once the schools for student sampling are selected, click on the button Draw Student Sample If a student sample was already drawn for a selected school, an alert message will appear on the screen (Figure 27). If you respond, Yes, KeyQuest will draw a new sample and overwrite the records in the Student Tracking Form related to this school. If you respond No, the program will not re-draw the sample for that school. 33

34 Figure 27: Warning message 4.4 CHANGING STUDENT TRACKING FORM LABELS 120. The Student Tracking instrument should be adapted correspondingly to the national list of study programmes. After student sample is drawn and before STF is printed, labels for STF should be modified. On the Sampling Screen (Figure 25). Click on Tracking forms button 121. The Student Tracking Form Screen will appear (Figure 28) 122. Double click on the school name(s) in the left window. The school name will move to the right window To change STF labels and list of study programmes: Select Initial report Click the Change Tracking Form Labels button Figure 28: Student Tracking Form Screen 124. Tracking Form Labels Screen will appear (Figure 29). 34

35 Figure 29: Tracking Form Labels Screen 125. Enter Number of study programmes as they appear in the second column of SPT (e.g. for Table 3 example this number is 2, because only two national programme names, Secondary school and Technical college are listed in this SPT) To change text < Programme 1>, < Programme 2>, to the programme names which are understood by the School Coordinators and Test Administrators: Scroll down the Tracking Labels window Click on <Programme 1> <Programme 1> text appears in the New Tracking Labels window. Insert national programme name for <Programme 1> as it appears in the SPT and in SC Manuals. Continue with all other programmes if the country has more than one programme of study. Change any other text of the tracking form as required, especially for foreign language requirements. Click Return to Previous button. 4.5 PRINTING AND UPDATING STUDENT TRACKING FORM After students are sampled and the labels and study programme list are modified as required, STF can be printed. On the Student Sampling Screen (Figure 25). Click on Print Tracking Form button The Student Tracking Form Screen will appear (Figure 26). Select Initial Report to print the STF for the School Coordinator. This report will include NO data for the SEN, Inclusion, and Participation Status variables. The School Coordinator will record SEN and Inclusion information prior to the testing If TA Report is selected, then data or default values will be printed for SEN and Inclusion variables, but not for Participation Status variables 130. If Final Report is selected, then data or default values will be printed for all variables Use the Data Entry Functionality of KeyQuest to update the data contained in the STF and to record the SEN, Inclusion, and Participation Status variables. It is a requirement to update these variables before the data entry process for the test booklets and questionnaires starts. KeyQuest will use this information to check identification variables. 35

36 5 DATABASE ADAPTATIONS 5.1 GENERAL COMMENTS 132. PISA is an international study, where countries with very different educational systems and cultural particularities participate. Due to this diversity, there is a need for countries to adapt some instruments. At the same time, the aim of PISA, as an international study, is to have a comparable international data from all participating countries. That is why the process of verification of national adaptations is an issue of the utmost importance for the Consortium. Please refer to the Chapter 3 (Exhibit 3-1) of the PISA 2003 Main Study NPM Manual for the timeline and instructions for the preparation of national version(s) of the Main Study test instruments and survey material While planning adaptations to your national instruments try to minimise and simplify possible changes. It is important to keep in mind that National adaptations should be seen from the aspect of comparability. National adaptations even if they are necessary endanger the comparability of the international data. That is why All national adaptations MUST be documented. All national adaptations MUST be verified and approved All adaptations done in the questionnaires have important implications in terms of data cleaning and data management. For this reason, the Questionnaire Adaptations Spreadsheet (QAS) containing the description of the adaptations that you are planning to introduce in the Questionnaires, must be submitted first for approval to - <pisa@acer.edu.au>. - Only when the Consortium approves changes are they permitted be implemented in the questionnaires When adaptations to the international PISA instruments are approved, you might need to make the matching changes in KeyQuest. These changes MUST be made BEFORE data entry begins. All changes made to the structure of pre-configured KeyQuest questionnaires must be recorded on the same Questionnaire Adaptation Spreadsheets (QAS) where adaptations to the questionnaires are approved. NOTE: All countries require some adaptations to the questionnaires. However, it is important to minimize and simplify changes as much as possible. Excessive adaptations make the verification and data cleaning processes unnecessary long and hence jeopardise the projects timeline This part of the document relates to the adaptations to the KeyQuest Questionnaire Instruments for the Main Study. Particularly, we explain how to record these adaptations on the QAS. Please refer to the Chapter 5 of the KeyQuest Manual for generic features of instrument management. 36

37 5.2 QUESTIONNAIRE ADAPTATIONS 5.2.daptations Required by the Consortium All terms and expressions that require adaptation in every country appear in the source version of questionnaires between <clamps>, with references to explanations and translation notes attached. These are terms like <ISCED>, <Programme>, <Grade>, <Mark>, etc. Please make sure that all these terms are translated into expressions that are actually used by the students and the teachers of your country rather than into technical terms used only by specialists. We refer to the questions that contain these terms as international categories The international categories <ISCED 1>, <ISCED 2> etc. should be replaced with the designations by which the countries refer to the study levels covered by those categories. Please conform to the international classification conventions adopted by your country (and described in detail in the OECD brochure Classifying Educational Programmes Manual for ISCED-97 Implementation in OECD Countries, 1999 Edition, OECD) The international category <grade> must be replaced with the current term used in your country to refer to the grade attended If in some or all schools the term <grade> differs from the international definition, please notify ACER about that in the <grade> section of QAS and in the SPT and give a translation of national <grade> into international terms. For example, national grade 1 (secondary school) translates into 8 th grade internationally. INTERNATIONAL DEFINITION OF <GRADE> USED IN PISA: Grade 1 is defined as the first year in ISCED 1 (Primary or Elementary Education). Any other school grade is defined as a number of years of schooling beginning from Grade 1. NOTE: It is the level of schooling that defines the grade and not necessarily the amount of time the student has spent at school. For example, a student who 'repeats' a year in school does not progress to another grade The term <mark> must be replaced with the current term used in your country. You also have a choice of options in the EC Questionnaire to collect information about students marks and a choice of variables in KeyQuest to enter them. Instructions on how to describe your choice in the relevant section of QAS is given in the Variable Modifications part of this section The international category <Programme>, meaning the Programme of Study for 15-yearold students in your country, also needs to be carefully adapted. This is explained in the Study Programme section of this document and in Chapter 5 of the PISA 2003 Main Study NPM Manual Other international categories (e. g. categories related to the language spoken at home and country of birth) also need adaptations. This is explained in the following sections When implementing such adaptations, please bear in mind that the national formulation selected for the international categories should both be easy to understand by the respondent AND, at the same time, enable the response-coding scheme to remain consistent across countries. To ensure that international categories are comparable, these adaptations must be submitted to ACER for prior approval before they are implemented. 37

38 5.2.2 National Options 145. It may also occur that the NPM wishes to delete or modify some items or to incorporate additional items into the School or Student Questionnaire for the purpose of national analyses. Should this be the case, please bear in mind that: All questions that the NPM may want to implement as national options must be submitted to ACER for prior approval; and The timing of the Student Questionnaire session will have to be modified accordingly, depending on the time the students will need to respond to any additional items Some of the changes in the questionnaires will need corresponding adaptations to the KeyQuest Instruments. All adaptations to the KeyQuest Instruments MUST be recorded on the QAS. The upgraded version of the QAS indicating both hard copy and software adaptations MUST be included in the data submission materials In the following sections we explain how to implement KeyQuest adaptations to the Questionnaires and how to record these adaptations on the QAS. Please refer to the Chapter 5 of the KeyQuest Manual for more generic features of instrument management Naming Convention International Version of Questionnaires 148. All item examples in this document are taken from the PISA 2003 Main Study questionnaires An item in this section refers to any single question of a questionnaire. For example, Q3 Is your school a <public> or a <private> school? (Please <tick> only one box.) A <public> school... 1 A <private> school A variable in this section refers to a representation of the item in KeyQuest. For example, this item is represented by the variable named SC03Q01 in KeyQuest. The same variable names are used in KeyQuest Codebooks and in the automatically generated SPSS and SAS control files when data are exported. 38