C:\Users\<your_user_name>\AppData\Roaming\IEA\IDBAnalyzerV3

Installing the IDB Analyzer (Version 3.1) Installing the IDB Analyzer (Version 3.1) A current version of the IDB Analyzer is available free of charge from the IEA website (http://www.iea.nl/data.html, see Figure 1 1 ). The size of the setup file is over 20MB. Additionally, the IDB Analyzer is also always available bundled together with the most recent IEA databases distributed on CD of DVD. To install the IDB Analyzer on your computer you will need to take the following steps: 1. Uninstall any previously installed versions of the IDB Analyzer 2. 2. Download or copy the installation file to a directory of your choice. 3. Double click on the IDB Analyzer installation file to start the installation process. 4. When installing the IDB Analyzer you will be prompted to select the language that you would like to use during the installation, you will be asked to accept the licensing agreement and to specify the destination folder where you want to install the application. We recommend you choose the default directory for the installation, suggested by the IDB Analyzer setup: C:\Users\<your_user_name>\AppData\Roaming\IEA\IDBAnalyzerV3 1 Actual look of this web page might have changed after the publication of this document. 2 If you have created syntax files with previous versions of the IDB Analyzer, and want to reuse these, you should NOT uninstall Version 2 of the IDB Analyzer. The IDB Analyzer (Version 3.1) page 1

5. Once the installation process is completed you are ready to use the IDB Analyzer. Figure 1: Screenshot of IDB Analyzer Download Page Installation Notes System Requirements for PC The IDB Analyzer will work on most IBM-compatible computers using Microsoft Windows XP or any later version. As the software no longer directly accesses the SPSS engine, SPSS for Windows does not need to be preinstalled on the machine, but it is needed to execute the SPSS code created by the IDB Analyzer. Recommended System Configuration: PC with 1 GHz or higher processor speed 512 megabytes (MB) of RAM or higher About 50 megabytes (MB) of available hard disk space during setup Super VGA (1024x768) or higher-resolution video adapter and monitor Keyboard and mouse or compatible pointing device Microsoft Windows XP, Vista, or 7 Microsoft Excel 2003 or later version SPSS for Windows Version 15 or later.net Framework 4.0: http://www.microsoft.com/enus/download/details.aspx?id=17718 The IDB Analyzer (Version 3.1) page 2

Preparing to Run the IDB Analyzer under Windows 7 When running Windows 7 on your machine, you will need to do the following before starting your SPSS session: 1. Make sure there are no SPSS windows open. 2. Go to the Start Menu. 3. Find the SPSS... entry. 4. Right Click on the SPSS... entry. 5. Select Properties Option. 6. Select the Compatibility Tab. 7. Under Privilege Level click the box for Run this Program as an Administrator. 8. Click Apply and then OK. You should now be ready to start the IDB Analyzer and use SPSS in your analysis. Preparing to Run the IDB Analyzer in a Mac Environment Currently there is no standalone Mac version of the IEA IDB Analyzer. However, the software can be used on Mac through a virtual machine and Windows (2000, XP, Vista or 7) installed on it. The current version was tested using Windows XP installed on Parallels Desktop for Mac (http://www.parallels.com/products/desktop/), although other virtual machine software, e.g. VirtualBox (https://www.virtualbox.org/) could also be used. In order to install a working copy of the IEA IDB Analyzer on a MAC, please follow these steps: 1. Install the virtual machine software on your Mac computer. 2. Install Windows (2000, XP, Vista or 7) on the virtual machine. 3. Install SPSS v15.0 or higher on the Windows installed on the virtual machine. 4. Install the IEA IDB Analyzer into the Windows installed on the virtual machine following the steps described in the previous section. Please note that running the Windows on a virtual machine can increase the hardware requirements of your Mac computer. Running the IEA IDB Analyzer, and the SPSS macros it produces, will be possible only in the Windows installed on the virtual machine. Starting the Application The IDB Analyzer consists of two separate modules integrated in a single application: the Merge Module and the Analysis Module. You can start the IDB Analyzer and access these modules by doing the following: START -> Programs -> IEA -> IDB AnalyzerV3 -> IDBAnalyzer Figure 2 shows the selection sequence for starting the application from the Start menu. The IDB Analyzer (Version 3.1) page 3

Figure 2: Starting the Application from the Start menu About the IDB Analyzer (Version 3.1) The IEA International Database Analyzer (IDB Analyzer) is an application developed by the IEA Data Processing and Research Center (IEA-DPC) in Hamburg, Germany, that can be used to combine and analyze data from IEA s large-scale assessments such as Trends in International Mathematics and Science Study (TIMSS), TIMSS Advanced, Progress in International Reading Literacy Study (PIRLS), the Second Information Technology in Education Study (SITES), the Teacher Education and Development Study in Mathematics (TEDS-M), the Civic Education Study (CivEd), and the International Civic and Citizenship Education Survey (ICCS) as well as analyze data from the Organisation for Economic Co-operation and Development s Teaching and Learning International Survey (TALIS), Programme for International Student Assessment (PISA), and Programme for the International Assessment of Adult Competencies (PIAAC). 3 The IDB Analyzer creates SPSS syntax that can be used to perform analysis with these international databases. It generates SPSS syntax that takes into account information from the sampling design in the computation of sampling variance. In addition, it generates SPSS code to handle the use of plausible values. The resulting code can be used to calculate estimates of achievement and their corresponding standard errors, combining sampling and imputation variance. The code generated by the IDB Analyzer enables the user to compute descriptive statistics and conduct statistical hypothesis testing among groups in the population without having to write any programming code. The IDB Analyzer is licensed free of cost, not sold, and is for use only in accordance with the terms of the licensing agreement. While you can use the software for free, you do not have ownership of the software itself or its components, including the SPSS macros, and you are only authorized to use the SPSS macros in combination with the IDB Analyzer unless explicitly 3 Upon request, the IDB Analyzer can be configured to work with other large-scale assessment databases. For further information please contact the IEA-DPC at software@iea-dpc.de. The IDB Analyzer (Version 3.1) page 4

authorized by the IEA. The license expires at the end of each calendar year, when you will again have to download and reinstall the most current version of the software. The IDB Analyzer consists of two modules the Merge Module and the Analysis Module which are integrated in one common application window. Support for the IDB Analyzer can be obtained by contacting the IEA Data Processing and Research center software unit at software@iea-dpc.de. The Merge Module The Merge Module is used to combine data files from different countries 4, and when necessary merge data files from different sources like student background questionnaires and achievement files, or student background files with teacher or school level files. It also allows the user to easily select individual or groups of variables to create a smaller and more manageable dataset. When running the Merge Module, the IDB Analyzer creates SPSS code that merges and combines files specified by the user, keeping only the selected variables. The SPSS code is automatically saved to an SPSS syntax file and opened in an SPSS syntax window. The data files created using the Merge Module can be processed either with the analysis module of the IDB Analyzer or by any other analysis software that accepts SPSS files as input. The Merge Module is only available to use with IEA databases and others in which the data are published separate by country, currently OECD-TALIS and OECD-PIAAC. The Merge Module is designed to combine data from the same assessment year. To combine data across 2 or more years, please refer to the appendices of this Help Manual. The Analysis Module The Analysis Module of the IDB Analyzer provides procedures for the computation of means, percentages, standard deviations, correlations, and regression coefficients for any variable of interest overall for a country, and for specific subgroups within a country. It also computes percentages of people in the population that are within, at, or above benchmarks of performance or within user-defined cut points in the proficiency distribution, percentiles based on the achievement scale, or any other continuous variable. The Analysis Module can be used to analyze data files from the above mentioned studies, regardless of whether they have been preprocessed with the IDB Analyzer Merge Module. The analysis module can create code for several analysis procedures. Like the merge module, the analysis module creates SPSS code that computes the statistics specified by the user. The following analyses can be performed with the Analysis Module: 1. Percentages and Means: Computes percentages, means, and standard deviations for selected variables by subgroups defined by the user. The percent of missing responses is included in the output. 2. Percentages only: Computes percentages by subgroups defined by the user. 4 The term country is used to refer to reporting jurisdictions in the corresponding study. These include countries per se, states, provinces, benchmarking participants, etc. Each study has adopted its own convention for naming and identifying these entities, but in general these are identified by a unique value in the variables usually called IDCNTRY, CNT, CNTRYID, etc. The IDB Analyzer (Version 3.1) page 5

3. Regression: Computes regression coefficients for selected variables predicting a dependent variable by subgroups defined by the user. New in this version of the IDB Analyzer is the capability of including plausible values as dependent or independent variables in the regression equation. When more than one set of plausible values are specified in the analysis, the analysis is carried out using the first of the plausible values, then the second, etc. In the end, the results are summarized across the plausible values. 4. Benchmarks: Computes percent of students meeting a set of user-specified performance or achievement benchmark by subgroups defined by the user. It computes these percentages in two modes: cumulative (percent of students at or above given points in the distribution) or discrete (percent of students within given points of the distribution). It can also compute the mean of an analysis variable for those at a particular achievement level. 5. Correlations: Computes correlation for selected variables by subgroups defined by the grouping variable(s). New in this version of the IDB Analyzer is the capability of computing the correlation between plausible values. When more than one set of plausible values are specified in the analysis, the analysis is carried out using the first of the plausible values, then the second, etc. In the end, the results are summarized across the plausible values. 6. Percentiles: Computes the score points that separate a given proportion of the distribution of scores by subgroups defined by the grouping variable(s). When calculating these statistics, the IDB Analyzer has the capability of using any continuous or categorical variable in the database, or make use of scores in the form of plausible values. When using plausible values, the IDB Analyzer generates code that takes into account the multiple imputation methodology in the calculation of the variance for statistics, as it applies to the corresponding study. All procedures offered within the analysis module of the IDB Analyzer make use of appropriate sampling weights and standard errors of the statistics that are computed according to the variance estimation procedure required by the design as it applies to the corresponding study. Before conducting data analysis with the IDB Analyzer, we recommend the user become familiar with the specifics of the study design of interest. Each study has its own Technical Report available online. What s New in the IDB Analyzer (Version 3.1)? Version 3.1 of the IDB Analyzer replaces Version 2.0 which will not be available or supported any longer. These two differ in several ways. Version 3.1 has a new look and incorporates new functionality not available in previous versions. New Look One integrated application: o Merge and Analysis Modules are combined into one integrated application (previously the Merge and Analysis Modules were separate applications). o All studies can be analyzed from the same application (previously there was a separate application for OECD-PISA analysis). Use of dropdown menus, instead of check boxes, for analysis specifications: analysis type, use of plausible values, benchmark options, missing data options, and number of decimals. The IDB Analyzer (Version 3.1) page 6

Streamlined presentation of analysis specification dropdown menus. Only those relevant to the selected statistic type are displayed on the screen (New in Version 3.1!). New analysis type selection scheme: by study and data aggregation level. New variable selection scheme: o New search capabilities by variable name and label. o Double click enabled to move variables between selection panels. o Expanding windows to accommodate longer lists of variables. Individually named output windows for each analysis. Fully labeled output for the correlation analysis. Separate output files for statistics computed with correlations and regression analysis. New Functionality Works with newer Windows Operating Systems and fully compatible with newer versions of SPSS. Requires SPSS 15 or later version. Fixes known bugs. Enhanced Analysis Features For all analysis and statistic types, the IDB Analyzer Version 3.1 adds some new features. Prior to every analysis, it calculates un-weighted and weighted descriptive statistics for the analysis variables (means, standard deviations, minimum and maximum), and frequencies by analysis subgroups. This is done to facilitate quality control of the input data. In addition, except when computing percentiles, the estimate of the population size for each of the subgroups processed (sum of the sampling weights) and the corresponding standard errors are computed. For the correlation analysis, Version 3.1 adds the following analysis features: Calculates correlations between plausible values. When more than one set of plausible values are specified in the analysis, the analysis is carried out using the first of the plausible values, then the second, etc. In the end, the results are summarized across the results from each of the plausible values. Calculates descriptive statistics for the data used (mean and standard deviation) Has the option of pairwise or listwise deletion of cases from the analysis No longer limits the number of variables to include in the correlation matrix For the regression analysis, Version 3.1 adds the following features: Calculates standardized regression coefficients and corresponding standard errors Calculates descriptive statistics for the data used in the analysis (mean, standard deviation, and variance) Calculates Analysis of Variance (ANOVA) statistics (sums of square regression, residual and total) The IDB Analyzer (Version 3.1) page 7

Calculates Model statistics (R Squared and Adjusted R Squared 5 ) Provides the option of pairwise or listwise deletion of cases from the analysis No longer limits the number of predictor variables used Allows plausible values to be used as dependent variables, independent variables, or both.. When more than one set of plausible values are specified in the analysis, the analysis is carried out using the first of the plausible values, then the second, etc. In the end, the results are summarized across the results from each of the plausible values. No longer limits the number of independent variables to use in the analysis It gives the user the option to dummy code or effect code a categorical independent variable to use in a regression analysis(new in Version 3.1!). Enhanced Output Features The layout of the Microsoft Excel output files has been modified to facilitate post-processing of the results, such as creation of charts and graphics using Excel or SPSS. In addition, titles and labels for the output have been fixed in several of the modules and subroutines. The IDB Analyzer Version 3.1 also automatically saves the SPSS syntax that is used for the merge or analysis. Fully labeled and better looking outputs for the correlation and regression analysis types are added separate output for the correlation and regression coefficients and descriptive statistics. Also, for the regression analysis type model, ANOVA statistical outputs were added. Last, it redirects the SPSS output from each separate analysis to its own output window, giving the user the option to save the SPSS output in a file format of his or her choosing. Bar and Line Charts for Percentages and Means Bar or line charts are now drawn by default when computing Percentages, Percentages and Means, and when calculating percent of the population within benchmarks with or without an analysis variable. Charts are drawn only when there are 3 or fewer grouping variables when computing percentages and means, and when there are 2 or fewer grouping variables when computing percents of the population within benchmarks with or without an analysis variable. Enhanced Interaction with SPSS As SPSS published new versions of the software, and the number of SPSS versions available to the IDB Analyzer users proliferated, it became increasingly difficult to ensure compatibility with all past, current, and future versions of SPSS. To avoid compatibility issues with future versions of SPSS, and to avoid interruptions due to SPSS upgrades, it was decided to limit Version 3.1 to creating the SPSS syntax necessary for combining files or conducting analysis, and leaving the user the option of running the syntax at a time of choice. It therefore creates the SPSS syntax and opens it within SPSS, but the user now has to submit the syntax to the SPSS engine for processing. This can be done with the keystrokes Ctrl+A followed by Ctrl+R, or by selecting the menu item Run and the option All to submit the code for processing. While Version 3.1 cannot be started from within SPSS, you no longer need to reinstall the IDB Analyzer when you install a newer version of SPSS. An additional advantage of this new interaction with SPSS is that the 5 The Adjusted R Squared statistic is calculated as [1 (1 R_Square) * (n 1) / (n p 1)], where p is the number of regressors and n is the sample size. While most statistical software used the actual number of cases for the value of n, SPSS uses the sum of the weights, thus resulting in different values for the Adjusted R Squared statistics. The IDB Analyzer (Version 3.1) page 8

syntax for the merge or the analysis is now automatically saved with a name, and in a directory of the choosing of the user. In addition to the changes described above, several of the SPSS macros used by the Version 3.1 have been optimized from their original version and are now able to process the analysis in less time and using fewer computing resources than previous versions. Differences between IDB Analyzer Version 3.0 and Version 3.1 The IDB Analyzer Version 3.1 was released 6 months after the release of Version 3.0. The following are the differences between Version 3.0 and Version 3.1: The Merge and Analysis Modules have been updated to recognize files from the TIMSS/PIRLS 2011 combined sample, and the TALIS 2013 administration When specifying a regression analysis, the user now has the option to create and use in the analysis dummy or effect coded variables with existing categorical variables. When specifying a benchmark analysis, the user has the option to include in the analysis cases that were not assigned plausible values. These cases are counted in the denominator of the calculations of the results. This option was developed specifically to handle the problem solving results in PIAAC, but is available for all databases. Menu selection in the analysis module has been streamlined so that only options relevant to the statistics type selected are displayed. All other existing functionality available in Version 3.0 has been preserved in Version 3.1. The IDB Analyzer (Version 3.1) page 9