TAR ACER. Test of Abstract Reasoning MANUAL

Transcription

1 TEST OF ABSTRACT REASONING TEST OF ABSTRACT REASONING TEST OF ABSTRACT REASONING TEST OF ABSTRACT REASONING TEST OF ABSTRACT ACER Test of Abstract Reasoning MANUAL TAR

2 ACER Press

3 First published 2007 by ACER Press Australian Council for Educational Research Ltd 19 Prospect Hill Road, Camberwell, Victoria, 3124 Copyright 2007 Australian Council for Educational Research All rights reserved. Except under the conditions described in the Copyright Act 1968 of Australia and subsequent amendments, no part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the written permission of the publishers. Only material contained in the photocopy masters may be reproduced, and then only in quantities sufficient for the purchaser s own proper use and not for resale to, or use by, any other person or organisation. Edited by Elisa Webb Cover design by R.T.J. Klinkhamer Typeset by Cannon Typesetting, Melbourne Printed by Print Impressions National Library of Australia Cataloguing-in-Publication data: Australian Council for Educational Research ACER test of abstract reasoning manual. Bibliography. For secondary students. ISBN ISBN Reasoning (Psychology) Testing Handbooks, manuals, etc. I. Title Visit our website:

4 Contents List of tables...v List of figures...vi List of appendices...vi About ACER Test of Abstract Reasoning (ACER TAR)... 1 Background...1 Description of the tests...1 Typical uses...2 Occupational uses...2 Educational and vocational guidance uses...2 Test Options... 3 Short or long form...3 Short Form (TAR45)...3 Long Form (TAR60)...3 Paper or online administration...3 Paper...3 Online...3 Scoring and Reporting... 4 Paper administration...4 ACER Test Scoring Services...4 Online Response Entry System...4 Online administration...5 Automated reporting...5 ACER Online Testing System...5 Direct request...5 Interpretation of ACER TAR Reports... 7 Interpreting norm scores...7 Percentile ranks...7 Stanine scores...7 Rasch scores...8 Comparison with Raven s Standard Progressive Matrices and Advanced Progressive Matrices...8 Case Studies Example 1: Recruitment for a public sector department...11 Example 2: Call centre selection...11 Example 3: Promoting floor staff to store manager...11 Directions for Administration General information...13 General information for online testing...13 Minimum system requirements...14 Specific instructions...14 Instructions for paper administration...15 Instructions for online administration...19 Technical Information Summary of research studies...22 Study 1 ACER TAR Short Form (TAR45)...23 Study design...23 Study sample...23 Testing procedure...24 iii

5 C o n t e n t s Item analysis...25 Reliability...25 TAR45 subsets...26 Correlations...26 Set correlations...26 Set reliability...27 Validity...27 Correlations with Raven s Standard Progressive Matrices...27 Scale scores...28 Demographic variables...28 Gender...28 Education level...29 Occupational classification...29 Language background...30 Age groups...30 Test order...31 Conclusion...31 Study 2 ACER TAR Long Form (TAR60)...32 Study design...32 Study sample...32 Testing procedure...33 Item analysis...33 Reliability...34 TAR60 subsets...34 Correlations...34 Set correlations...34 Set reliability...35 Validity...36 Correlations with Raven s Advanced Progressive Matrices...36 Set correlations...36 Calculating the scale scores...37 Equating TAR60 to APM...37 Background variables...38 Gender...38 Education level...38 Language background...39 Study type...39 Test order...40 ACER TAR Short Form (TAR45)...40 Background variables...40 Gender...40 Education level...41 Language background...42 Study type...42 Test order...43 Conclusion...43 Study 3 ACER TAR Short Form (TAR45) Study design...44 Reliability...44 Gender differences...45 Conclusion...46 References...47 Appendices...48 iv

6 List of Tables Table 1 Source of items in ACER TAR...1 Table 2 Example occupation groups...2 Table 3 Examples of appropriate forms for selected jobs...3 Table 4 Stanine groups and descriptors...7 Table 5 Simple conversion table between ACER TAR45 and Raven s SPM...9 Table 6 Simple conversion table between ACER TAR60 and Raven s APM...10 Table 7 Key statistical findings in Study Table 8 Key statistical findings in Study Table 9 Key statistical findings in Study Table 10 Demographics Gender...23 Table 11 Demographics Highest level of education completed...23 Table 12 Demographics Occupational classification...24 Table 13 Demographics Age group...24 Table 14 Demographics Language spoken at home...24 Table 15 Demographics Order of tests...25 Table 16 TAR45 set descriptions...26 Table 17 TAR45 internal set correlations...26 Table 18 TAR45 set correlations with total score...26 Table 19 TAR45 set reliabilities...27 Table 20 TAR45 mean and standard deviation by gender...28 Table 21 ANOVA TAR45 scores on gender...28 Table 22 TAR45 mean and standard deviation by education level...29 Table 23 ANOVA TAR45 scores on educational level...29 Table 24 TAR45 mean and standard deviation by occupational classification...29 Table 25 ANOVA TAR45 scores on occupational classification...30 Table 26 TAR45 mean and standard deviation by language...30 Table 27 ANOVA TAR45 scores on language...30 Table 28 TAR45 mean and standard deviation by age groups...30 Table 29 ANOVA TAR45 scores on age groups...31 Table 30 TAR45 mean and standard deviation by test order...31 Table 31 ANOVA TAR45 scores on test order...31 Table 32 Demographics Study types...32 Table 33 Demographics Highest level of education completed...32 Table 34 Demographics Gender...33 Table 35 Demographics Language spoken at home...33 Table 36 Demographics Order of tests...33 Table 37 TAR60 set descriptions...34 Table 38 TAR60 internal set correlations...35 Table 39 TAR60 set correlations with total score...35 Table 40 TAR60 set reliabilities...35 Table 41 TAR60 set correlations with APM total score...36 Table 42 Correlations of TAR60 items with APM total score...36 Table 43 TAR60 mean and standard deviation by gender...38 Table 44 ANOVA TAR60 scores on gender...38 Table 45 TAR60 mean and standard deviation by education level...38 Table 46 ANOVA TAR60 scores on educational level...39 Table 47 TAR60 mean and standard deviation by language...39 Table 48 ANOVA TAR60 scores on language...39 Table 49 TAR60 mean and standard deviation by study types...39 Table 50 ANOVA TAR60 scores on study types...40 v

7 L i s t o f Ta b l e s Table 51 Table 52 Table 53 Table 54 Table 55 Table 56 Table 57 Table 58 Table 59 Table 60 Table 61 Table 62 Table 63 Table 64 Table 65 Table 66 Table 67 Table 68 TAR60 mean and standard deviation by test order...40 ANOVA TAR60 scores on test order...40 TAR45 mean and standard deviation by gender...41 ANOVA TAR45 scores on gender...41 TAR45 mean and standard deviation by education level...41 ANOVA TAR45 scores on education level...41 TAR45 mean and standard deviation by language...42 ANOVA TAR45 scores on language...42 TAR45 mean and standard deviation by study types...42 ANOVA TAR45 scores on study types...43 TAR45 mean and standard deviation by test order...43 ANOVA TAR45 scores on test order...43 Number of males and females in the reference group...44 Descriptive statistics for TAR Reliabilities of TAR45 subscales...45 Correlations between subsets in TAR TAR45 scores for males and females...45 Statistics from independent sample t tests...46 List of Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Example of a group report from ACER Test Scoring Services...4 Example of an automated report...5 Example of a group report from the Practitioner s site...6 Bell-shaped distribution % confidence belt for equating between TAR45 logits and SPM logits % confidence belt for equating between TAR60 logits and APM logits...37 List of Appendices Appendix A QUEST item analysis for all 60 TAR items (Study 1)...48 Appendix B QUEST output for TAR45 (Study 1)...50 Appendix C Detailed conversion table between TAR45 and SPM (Study 1)...52 Appendix D QUEST item analysis for all 60 TAR items (Study 2)...53 Appendix E QUEST output for TAR45 (Study 2)...55 Appendix F Detailed conversion table between TAR60 and APM (Study 2)...57 vi

8 About ACER Test of Abstract Reasoning (ACER TAR) Background ACER Test of Abstract Reasoning (ACER TAR) has been developed in response to demand for an abstract reasoning assessment with Australian adult data. The items from ACER TAR have been sourced from two existing, well-researched ACER tests of abstract reasoning: the Middle Years Ability Test (MYAT) and the Aptitude Profile Test Series (APTS). Items from these two existing tests have been combined to create the ACER TAR Long and Short forms: Table 1 Source of items in ACER TAR Test form ACER TAR Short Form (TAR45) ACER TAR Long Form (TAR60) Item source 45 abstract reasoning items from MYAT 45 abstract reasoning items from MYAT 15 abstract reasoning items from APTS Items were chosen to reflect an appropriate range of difficulty and task sub-type. The data samples detailed in this manual are based on: (a) a study conducted on 151 adults in Australia who completed both ACER TAR and the Raven s Standard Progressive Matrices (SPM); (b) a study conducted on 295 adults at an Australian university who completed both ACER TAR and the Raven s Advanced Progressive Matrices (APM); and (c) a study conducted on 329 adults seeking work who had completed ACER TAR online. Further research studies and analyses of ongoing data collection projects will be reported in future editions of this manual. Description of the tests ACER TAR is a test of abstract (or non-verbal) reasoning. The MYAT Manual defines abstract (non-verbal) reasoning as the extent to which [a person] can discover principles and rules, and apply them to solve problems using abstract visual patterns rather than numbers and words (p. 1). The APTS Manual defines abstract reasoning as the ability to discover the principle/s [that] determine the rule or rules that govern the progression of a pattern from one diagram to another in a series, or to identify the part which is missing from a diagram (p. 3). Tests of abstract reasoning such as the ACER TAR are nearly language-free and so may avoid problems associated with the use of verbal reasoning tests in some contexts where candidates written English skills are not well developed, such as where candidates are from non-english speaking backgrounds or have specific language learning difficulties. Statistical analyses detailed in this manual (see Technical information, p. 22) confirm that language background is not a significant source of variance in performance on the ACER TAR.

9 A C E R Te s t o f A b s t r a c t R e a s o n i n g Typical uses Occupational uses ACER TAR is typically used as a measure of general ability for selection into occupations which involve a moderate to high level of demand on reasoning ability and for other purposes where the ability to think clearly, analyse information, solve problems and make rational decisions are important. Examples of selection criteria linked to measures of abstract reasoning are: Planning and coordinating activities Creating innovative solutions to problems Making decisions in a rational manner Learning new material quickly Collecting and analysing information (Power, 2004, p. 24) Table 2 provides examples of occupational groups for which the ACER TAR can be used. Table 2 Example occupation groups Main occupation group Managers and administrators Professionals Para-professionals Tradespersons Artistic and creative Figures and computational Literary Medical Technical and engineering Scientific Examples of occupations General and specialist managers, managing supervisors, business managers Scientists, architects, engineers, medical practitioners, school teachers, social and business professionals Medical and science technical officers, technicians, technical workers, registered nurses, police officers Fitters, mechanics, electrical and electronic tradespersons Graphic artists, photographers, pianists, actors, dancers, desktop publishers Accountants, clerks, data analysts, computer programmers, mathematicians, physicists Authors, journalists, editors, script writers Surgeons, general practitioners, nurses Electrical technicians, mechanical engineers, architects Biologists, chemists, physicists, mathematicians, surveyors, teachers (APTS, 2000, pp. 6 7) Educational and vocational guidance uses The ACER TAR tests may also be used in conjunction with a record of attainment and other data, in counselling a candidate in selecting an appropriate course of study, and in the selection of occupations where further study and training are involved. (See Nelson Bolles, 2002; Holland, 1997; Kummerow, 1991; Anastasi, 1976; Cronbach, 1970.)

10 Test Options Short or long form ACER TAR has two forms: a Short Form (TAR45) and a Long Form (TAR60). Short Form (TAR45) TAR45 is the easier of the two tests and is intended for use with candidates who have completed at least Year 11 and who are applying for a variety of technical, clerical and administrative positions where in-service training, or part-time or after work study is involved. Long Form (TAR60) TAR60 includes the 45 items in the Short Form, plus an additional 15 items. TAR60 is the more challenging test and is appropriate for use with candidates who have completed at least Year 12 and are applying for positions where the work requires a high level of reasoning ability or where these qualities will become increasingly important as staff are promoted in a career system. Table 3 provides examples of appropriate forms for selected jobs. Table 3 Examples of appropriate forms for selected jobs Short Form (TAR45) Personal assistant Market research interviewer Accounts clerk Insurance investigator Customer service operator Call centre operator Workplace trainer Legal clerk Long Form (TAR60) Journalist/copywriter Computer programmer/analyst Market research analyst Financial advisors Graduates Managers Executives Paper or online administration ACER TAR is available for online administration or administration in paper and pencil format. In both cases ACER TAR should always be conducted under appropriate supervision. In no circumstances should ACER TAR be administered in an unsupervised environment. Candidates should not be provided with login codes to access the online test from home or other remote locations unless appropriate supervision has been put in place. Paper Test administrators should provide one ACER TAR reusable test booklet and one ACER TAR answer sheet for each candidate. Completed ACER TAR OMR answer sheets should be sent to ACER Test Scoring Services for scoring and report generation. Alternatively, responses on non-omr answer sheets can be keyed into the Online Response Entry System by the test administrator for immediate access to results. Online Test administrators should provide each candidate with a unique login code, which allows candidates to access the tests online. On completion of the online test, results will be automatically generated and a report ed to the test administrator. Further details regarding online testing are provided in Directions for Administration on p. 13. Contact ACER Press Customer Service on for more information about setting up an online testing account.

11 Scoring and Reporting There are no score keys or norm tables in this manual (apart from those relating to conversion between ACER TAR and Raven s tests on pp. 9 10). ACER TAR can only be scored through one of the automated scoring systems described below. Paper administration ACER TAR can be administered in paper and pencil format using the ACER TAR test booklet and an ACER TAR answer sheet. There are two options for scoring and reporting. ACER Test Scoring Services Information on how to use the ACER Test Scoring Services can be found at ACER TAR OMR answer sheets sent in for test scoring should be accompanied by a copy of the Test Scoring Order Form that comes with orders of the answer sheet. Reports will be provided to test administrators in a format similar to the report shown in Figure 1 below. Candidates can be listed alphabetically or based on test scores. Figure 1 Example of a group report from ACER Test Scoring Services Online Response Entry System Test administrators can manually enter data from non-omr administrations into the online system. ACER TAR Online Response Entry login codes can be purchased from ACER Press Customer Service. Once the response data has been confirmed, the online system will automatically score the responses and generate a report. See the Online administration section (p. 5) for more details on getting access to candidate results.

12 S c o r i n g a n d R e p o r t i n g Online administration Tests completed online are automatically scored by the system. Reports can be obtained in one of three ways: automated reporting; via the Practitioner s site of the ACER Online Testing System; or direct request to ACER Press Customer Service. Automated reporting To obtain automated reports, an address must be supplied at the time of purchasing ACER TAR login codes. The address supplied should be that of the test administrator or HR manager, not the candidate. A separate report (see Figure 2) is generated for each ACER TAR candidate and automatically sent to the test administrator once the responses are submitted online. Figure 2 Example of an automated report ACER Online Testing System Test administrators can set up an Online Testing System account by contacting ACER Press Customer Service on Once this is done, individual or group results can be accessed at any time from the Practitioner s site. An example of how results are displayed is shown in Figure 3 (p. 6). Direct request Test administrators who are conducting large-scale testing may prefer to wait until the last candidate has completed testing and then request ACER Press Customer Service to compile the results of all candidates in the group in a single report. Direct request reports are also available for test administrators using the Online Response Entry System for scoring and reporting of paper tests.

13 A C E R Te s t o f A b s t r a c t R e a s o n i n g Figure 3 Example of a group report from the Practitioner s site.

14 Interpretation of ACER TAR Reports The raw score on a general ability test has little meaning in itself. Usually the first step in interpreting performance is to convert the number of correct answers (raw scores) into terms that will allow for comparison with a reference group who have taken the test (normative data). Normative data for TAR45 have been obtained from a sample of 151 adults in Australia. See Study design, p. 23 for more information about the sample. Normative data for TAR60 have been obtained from a sample of 295 adults at an Australian university. See Study design, p. 32 for more information about the sample. Converted scores in the ACER TAR reports are presented in terms of percentile ranks, stanines and Rasch scale scores. Interpreting norm scores Norm scores such as percentile ranks and stanines (see below) for use with selection instruments such as ACER TAR enable recruitment officers to place candidates in order of merit, and to select an appropriate cut off point below which candidates will be excluded from further consideration. (See Case studies, p. 11 for examples of how this can be done.) Percentile ranks Percentile ranks show the percentage of the reference group with scores below the candidate s score. For example, a percentile rank of 63 means that the candidate has performed better than 63 per cent of the candidates in the reference group; a percentile rank of 23 means that the candidate has performed better than only 23 per cent of the candidates in the reference group. While percentile ranks are useful as a means of locating a candidate s standing relative to the norm group, one drawback is that percentile ranks are not evenly spaced along a measurement scale. Percentile ranks for a large, representative sample tend to be bunched up somewhere around the middle of a bell-shaped distribution, with smaller numbers of candidates towards the upper and lower ends of the distribution. Because small changes in raw scores around the middle of the distribution will have a considerable impact on the percentile rank, it is easy to over-emphasise differences near the middle set of scores and to under-emphasise differences near the extremes where very few candidates are located. Stanine scores To help overcome the problem of using a unit that is not evenly spaced along the measurement scale, stanine scores have been included. In order to derive stanine scores, nine categories are used. Stanines (except for stanines 1 and 9) are of equal length along the variable axis underlying the normal curve. A stanine of 9 indicates a very high level of achievement relative to the reference group while a stanine of 1 indicates a very low relative achievement. Table 4 shows the percentage of the reference group assigned to each stanine score and a descriptor. Table 4 Stanine groups and descriptors Stanine Percentage of group Descriptor 9 4 Very high Above average Average Below average 1 4 Very low

15 A C E R Te s t o f A b s t r a c t R e a s o n i n g Figure 4 Bell-shaped distribution Rasch scores Raw scores (number of correct scores) on the ACER TAR tests are not on an interval scale. In other words, equal numerical differences between raw scores do not represent equal differences in ability. To provide scores on an interval scale, raw scores have been transformed to scale scores using the Rasch measurement model. Measurement errors associated with these scale scores are also reported. The resulting scale is called the TAR Scale. The development of the TAR Scale is outlined in Scale scores, p. 28. Comparison with Raven s Standard Progressive Matrices (SPM) and Advanced Progressive Matrices (APM) The provisional data from the various studies in Technical Information on p. 22 included a comparison of ACER TAR with the Raven s SPM and APM. Rasch modelling of the data collected has allowed the construction of a table for approximate comparisons between TAR45 raw scores and SPM raw scores as well as TAR60 raw scores and APM raw scores. Raw scores on the SPM and APM can be used to generate percentiles and stanines on TAR45 and TAR60 using the norm data available, and vice versa. Table 5 is a simplified version of the full table comparing TAR45 and SPM raw scores that can be found in Appendix C.

16 I n t e r p r e t a t i o n o f A C E R TA R R e p o r t s Table 5 Simple conversion table between ACER TAR45 and Raven s SPM TAR45 raw score TAR45 scale score SPM raw score * Numbers in bold are obtained by linear interpolation

17 A C E R Te s t o f A b s t r a c t R e a s o n i n g Table 6 is a simplified version of the full table comparing TAR60 and APM raw scores that can be found in Appendix F. Table 6 Simple conversion table between ACER TAR60 and Raven s APM TAR60 raw score TAR60 scale score APM raw score

18 Case Studies Example 1: Recruitment for a public sector department A recruiting officer for a public sector department has been put in charge of the graduate selection program. There are 100 initial applicants and the recruiting officer s aim is to reduce that number to approximately 25 for the next stage. Knowing that successful applicants will need to demonstrate a high level of general ability, she decides to screen all applicants with TAR45. The recruiting officer has access to a computer training lab with 25 computer terminals. She arranges for all applicants to attend one of four testing sessions over two days. After the last testing session is completed, she contacts ACER Press Customer Service and requests a group report on the results of the 100 applicants. The spreadsheet is ed to her showing the candidates achievement by percentile rank, stanine and scale score. Using the spreadsheet, the recruiting officer orders the list by percentile rank. In scanning down the list from highest percentile rank to lowest, she notes that selecting a cut off score of the 80th percentile would retain 27 candidates. This number is in line with the number of candidates the recruiting officer was hoping to select for the next stage, and confirms that all the selected candidates demonstrate a high level of general ability. Example 2: Call centre selection A recruitment agency has been asked to fill five vacancies for a special marketing project in a call centre. On advertising the position, 200 suitable applications are received. This number is reduced to 50 by the recruitment agency consultant on reviewing the applicants CVs. The consultant speaks with the manager of the call centre about the job. In their discussion the consultant notes that the level of detail required for this project is relatively high, the training window is relatively short and that information will need to be relayed to customers in a clear, logical and informed manner. The consultant concludes that the successful applicant will need to have a moderately high level of general reasoning ability. The manager indicates that she would like to have the applicants undergo some ability testing and attend a group interview in which various role-play activities will be conducted. From this exercise, 10 applicants will be selected for individual interviews. The consultant invites the 50 applicants to attend the group interview. Before starting the role-play activities, he administers TAR45 to all the applicants in paper and pen format. While the group interviews are being conducted, the consultant directs his assistant to enter the candidates responses into the ACER Online Response Entry System. When all responses are entered, the assistant accesses the ACER Online Testing System and downloads a group report on the results of the 50 applicants. The assistant uses the report to rank the candidates by achievement, and gives the spreadsheet to the consultant. At the completion of the group interview, the consultant makes a list of the 10 most likely candidates for the next stage and a further 10 possibles. He compares this list against the results of the test. He notes that of the list of most likely candidates, two have performed poorly or very poorly on the test, with percentile ranks of 28 or below. The other eight have performed well or very well, with percentile ranks above 70. He looks at the scores of the next list of possibles and sees that two of them have performed very well on the test with percentile ranks above 70. He adds these to the list of most likely candidates for further interviewing. Example 3: Promoting floor staff to store manager A trucking company is looking to promote one of the warehouse floor staff to the position of store manager. In addition to general organisational skills, the store manager must be able to work with basic mathematical concepts in areas such as stock control. Five of the warehouse staff apply for the position, and the general manager decides to interview them all. Three of the five applicants are from a non-english speaking background. A high level of English language proficiency is not required, although an assessment of the candidates general ability would be useful. The HR manager decides 11

19 A C E R Te s t o f A b s t r a c t R e a s o n i n g to administer two tests of general ability: ACER Numerical Select for assessing numerical reasoning skills, and the language-free ACER TAR for assessing general ability. The HR manager asks each of the candidates to come into the office in turn to complete the test session on the computer. Both tests are completed online. As each of the candidates completes the test, the results are automatically generated and ed to the HR manager. When all five applicants have completed the testing, the HR manager transfers their results (percentile ranks and stanines) to a spreadsheet and orders them first by ACER Numerical Select results and then by ACER TAR results. Of the five applicants, two score low or very low, at stanine 3 or below, on both tests. Two of the applicants score well on both tests, at stanines 7 or 8, showing both above average numerical reasoning and abstract reasoning ability. The last candidate scores an average stanine 5 on ACER Numerical Select, but stanine 9 (a very high result the highest of the group) on ACER TAR. The HR manager points out to the general manager that while this last candidate scored significantly lower than the other two on numerical reasoning, the ACER TAR score indicates a very high level of abstract reasoning. This is indicative of the ability to learn new things quickly and engage in strategic thinking. Concluding that specific skills in analysing data can be taught, the general manager and HR manager decide to promote the person with the high ACER TAR score and provide him with training in data analysis using their inventory tracking software. 12

20 Directions for Administration General information ACER TAR Short Form (TAR45) has 45 items and should be completed in 20 minutes. ACER TAR Long Form (TAR60) has 60 items and should be completed in 25 minutes. Please note these details and ensure that the instructions given to candidates are accurate. The data for the interpretation of the test were collected using standardised instructions. It is important that the examiner does not depart from these instructions when administering the test. To achieve this, the following points should be carefully observed: When administering these tests, do not vary in any way from the instructions given in the manual. No candidate should see a test booklet before testing time, and there should be no opportunity for a candidate to see any test booklet other than his/her own, during the testing session. All test booklets should be accounted for at the completion of testing. These are timed tests; it will be necessary to have either a stop watch or an ordinary watch with a second hand. If the latter is used, the starting and finishing time of the testing session should be recorded. The time limits given for the working of each section of the test should be observed exactly. If a test candidate asks a question during the practice items, it should be dealt with as follows: (a) If the question concerns details of procedure, e.g. Where do I write the answers?, it should be answered directly. (b) If the question concerns the explanation of an answer, the relevant instructions should be repeated. Paraphrasing should be used with great caution to ensure that the explanation does not differ in any substantial way from that given in the instructions. (c) Once candidates have started work on the actual test paper, no questions other than procedural questions should be answered. To ensure that the candidates perform at their best, the anxiety level of the candidates should be minimal. Therefore while adhering strictly to the instructions, try to avoid any feeling of strain or strangeness. The physical comfort of the candidates should be given careful consideration; the testing environment should have adequate seating, work space, lighting and ventilation. All practical arrangements should be considered well before the testing session. The following points should be noted: (a) The tester should be familiar both with the directions for administration and with the relevant test booklet well before the testing session. (b) A place for testing should be arranged and the seating and room layout checked. (c) Adequate quantities of test materials should be available. (d) If more than 30 candidates are to be tested, an assistant should be present. (e) Although candidates usually provide their own pens/pencils, the tester is advised to provide a few spares. (f ) Measures should be taken to ensure that external noise is minimal and that the testing session will be completed without interruption. General information for online testing All practical arrangements should be considered well before the testing session. The following points should be noted: A place for testing should be organised with a sufficient number of computers arranged for testing purposes (so that, for example, the test administrator can see all screens, candidates can only see their own screen, etc.) All computers should be checked (screen, keyboard, mouse, browser settings, internet connection, etc.) prior to the testing session. 13

21 A C E R Te s t o f A b s t r a c t R e a s o n i n g If required, an assistant should be present to monitor candidates. This is essential if the number of candidates is above 20. Paper and pencils for working should be provided for each candidate. Measures should be taken to ensure that external noise is minimal and that the testing session will be completed without interruption. During the test sessions, candidates should not use any other program or navigate to any other website other than the testing website. In addition, candidates should only use the buttons in the browser window (e.g. NEXT, PREVIOUS) and not use the web browser s Back or Forward buttons. Consideration should be given to removing software and/or access to programs and websites on the computers that will allow candidates to access information that would assist them in answering the test questions (e.g. calculators, dictionaries, general internet access). In any case, candidates should be closely monitored to ensure that they do not navigate away from the online testing window. Minimum system requirements Users will need access to computers with Internet Explorer version 6 or greater. AutoComplete for forms should be disabled. Sufficient internet connection bandwidth should be available at least 10 kb per second per candidate. This should be checked with the administrator of the internet/network. The requirements for ACER TAR are slightly higher than text-based tests due to the use of image files. In general terms: a single candidate should have the equivalent bandwidth (at a minimum) of a dial-up connection 20 candidates should have the equivalent of a broadband connection (512K). Specific instructions The detailed instructions for paper and online administrations set out exactly what should be done and said. The spoken instructions are enclosed in boxes and should be given verbatim. Do not say anything other that what is indicated in the boxes. 14

22 D i r e c t i o n s f o r A d m i n i s t r a t i o n Instructions for paper administration Make sure all candidates have a pencil, test booklet and answer sheet. Say: The test you will be doing today is a test of observation and problem solving using patterns, shapes and diagrams. On your desk there should be a test booklet and an answer sheet. Hold up the test booklet. Say: This is the test booklet. Please do not open the test booklet until you are told to. Hold up the answer sheet. Say: This is the answer sheet. All your answers are to be marked on the answer sheet. Do not make any marks on the test booklet. Say: The answer sheet you are using will be scored by computer. It is important that you mark it properly. You will see that on the answer sheet there are many oval shapes or bubbles with letters or numbers in them. To give your answers to the test, you need to fill in these bubbles with a pencil. You should fill in the bubble completely, not just mark it with a tick, line, cross or circle. If you change your mind about an answer, you should completely erase the mark you have made and fill in another bubble. To practice using the answer sheet, we will all now fill in our personal details in the top half of the sheet. Hold up the answer sheet. Say: Where the answer sheet says Name, write in your name. Now look across the sheet and print your surname in the squares under SURNAME/FAMILY NAME. Now fill in the bubbles underneath. If your surname is Eade, you would fill in the bubbles E, then A, then D, then E. Now print your initials in the squares under INITS. Now fill in the bubbles underneath. Check that the candidates are correctly filling in the answer sheets. Say: Where the answer sheet says TEST FORM, fill in the bubble for (Short Form/Long Form). Say: Now fill in your sex and date of birth. Please note that this information will be used solely for identification and research purposes. They will not be reported in your results. Wait for a few seconds. Hold up the test booklet. Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

23 A C E R Te s t o f A b s t r a c t R e a s o n i n g Say: We are now ready to look at the test booklet and do some practice questions. Please open your test booklet to the first page. Do not turn the page until I tell you to. Hold up the test booklet open to the first page. Say: Look at the first practice question P1. The instruction says: The three shapes in the top row are alike in some way. Which shape in the bottom row is most like them? You will see that the five shapes in the bottom row each have a letter above them. On your answer sheet, look for the shaded rectangle that says Practice Examples. Next to P1, fill in the bubble with the letter of the shape you think is most like the top three shapes. Wait for a few seconds. Say: The correct answer is shape A, so you should have filled in the bubble A next to P1 on the answer sheet. Remember not to mark the test booklet, only the answer sheet. Now have a look at the second practice question P2. The instruction says: On the top row there are five squares with shapes arranged in order. One shape is missing. Which shape from the bottom row best completes the pattern? On the answer sheet next to P2 fill in the bubble with the letter of the shape you think best completes the pattern. Wait for a few seconds. Say: The correct answer is shape E, so you should have filled in the bubble E next to P2 on the answer sheet. Now have a look at the third practice question P3. The instruction says: There is one missing shape in the pattern at the top. Which shape from the bottom row best completes the pattern? On the answer sheet next to P3 fill in the bubble with the letter of the shape you think best completes the pattern. Wait for a few seconds. Say: The correct answer is shape A, so you should have filled in the bubble A next to P3 on the answer sheet. That completes the practice questions. A few things to remember: Fill in your answer on the answer sheet, not the test booklet When you choose an answer, fill in the whole bubble on the answer sheet don t tick, cross or circle the answer 16 Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

24 D i r e c t i o n s f o r A d m i n i s t r a t i o n If you change your mind about an answer, erase the mark you have made and fill in the bubble of your new choice. Are there any questions? Answer any questions about filling in the answer sheet. If you are administering TAR45, say: In the test booklet there are 60 questions. Today we are only doing the first 45 questions. You should finish at question 45 where it says END OF SECTION 1. Hold up the answer sheet and point to the bottom half of the sheet where the candidates fill in their answers. Say: You can see on the answer sheet that there is space for answers to 60 questions. We will only fill in the first 45 down to where it says END OF SECTION 1. Are there any questions? If you are administering TAR60, say: In the test booklet there are 60 questions. After 45 questions, there is a message END OF SECTION 1. Please ignore that message and work through the test booklet all the way to the last page. Answer any questions about the number of items to be completed. Hold up the test booklet. Say: Pause. In Section 1 of the test booklet there are five sets of questions. The instruction for the questions is given once at the beginning of the set. So for each instruction there are several questions. Each new set of questions has a new instruction for all the questions in that set. If you are administering TAR45, say: You will have 20 minutes to complete all the questions. Please note that there should be no talking during the test. If you need anything or want to ask a question, just raise your hand and one of the supervisors will help you. Don t spend too much time on any one question try to answer as many questions as you can in the time. After 20 minutes I will tell you to stop writing. Please put down your pencils and wait for me to collect your answer sheet. Before we begin, are there any questions? If you are administering TAR60, say: You will have 25 minutes to complete all the questions. Please note that there should be no talking during the test. If you need anything or want to ask a question, just raise your hand and one of the supervisors will help you. Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

25 A C E R Te s t o f A b s t r a c t R e a s o n i n g Don t spend too much time on any one question try to answer as many questions as you can in the time. After 25 minutes, I will tell you to stop writing. Please put down your pencils and wait for me to collect your answer sheet. Before we begin, are there any questions? Answer any questions. Say: We are now ready to begin the test. You have (20/25) minutes to complete the test, please turn the page now and begin. If administering TAR45, wait 20 minutes. If administering TAR60, wait 25 minutes. When the time limit is up, say: Please stop writing and put your pencils down. Collect the test booklets and answer sheets. Make sure that each candidate has correctly filled in their details on the answer sheet. 18 Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

26 D i r e c t i o n s f o r A d m i n i s t r a t i o n Instructions for online administration Make sure all candidates have a login code and a pencil and some paper for working. All the candidates computers should already display the ACER Press Online Testing System login page. Say: Please enter your login code and click the ENTER button. or you may enter the login codes into the screens and click the ENTER button prior to the candidates arrival. Say: Please fill in the personal details requested. Do not click submit until you are told to do so. Please note that the personal details requested are used for identification and research purposes only and will not be provided along with test results. Allow time for filling in the personal details. When all candidates have filled them in, say: Now click the SUBMIT button (Pause). The test you will be doing today is a test of observation and problem solving using patterns, shapes and diagrams. If you are administering TAR45, say: Look at the screen and follow while I read. This test is a timed test and will take 20 minutes. There are 45 questions in this test. There are some practice questions to complete before the test begins. If you are administering TAR60, say: Look at the screen and follow while I read. This test is a timed test and will take 25 minutes. There are 60 questions in this test. There are some practice questions to complete before the test begins. Say: Now click on BEGIN PRACTICE QUESTIONS. On the next few screens you will see a number of practice questions to help you in the test that follows. All the questions require choosing one answer from the list of possible answers. To answer each question, click the small circle below the answer. Look at the first practice question. The instruction says: The three shapes in the top row are alike in some way. Which shape in the bottom row is most like them? You will see that the five shapes in the bottom row each have a small circle below them. Click on the circle under the shape you think is most like the top three shapes. The circle should now have a dark spot inside it. Wait for a few seconds. Say: The correct answer is the first shape, so you should have clicked the small circle under the first shape. Now click NEXT and have a look at the second practice question. The instruction says: Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

27 A C E R Te s t o f A b s t r a c t R e a s o n i n g On the top row there are five squares with shapes arranged in order. One shape is missing. Which shape from the bottom row best completes the pattern? Click the small circle under the shape you think best completes the pattern. Wait for a few seconds. Say: The correct answer is the fifth shape, so you should have clicked the small circle under the fifth shape. Now click NEXT and have a look at the third practice question. The instruction says: There is one missing shape in the pattern at the top. Which shape from the bottom row best completes the pattern? Click the small circle under the shape you think best completes the pattern. Wait for a few seconds. Say: The correct answer is the first shape, so you should have clicked the small circle under the first shape. That completes the practice questions. Click NEXT and wait. Do not click ahead until I tell you to. Say: In the test there is an instruction for each of the questions. Several questions in a row will have the same instruction, then there will be a change of instruction. Be careful to note when the instruction changes. If you are administering TAR45, say: You will have 20 minutes to complete all the questions. If you are administering TAR60, say: You will have 25 minutes to complete all the questions. Say: Please note that there should be no talking during the test. If you need anything or want to ask a question, just raise your hand and one of the supervisors will help you. Don t spend too much time on any one question try to answer as many questions as you can in the time. Before we begin, are there any questions? Answer any questions. Say: We are now ready to begin the test. 20 Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

28 D i r e c t i o n s f o r A d m i n i s t r a t i o n During the test, use only the NEXT and PREVIOUS buttons under each question or the numbered navigation panel on the left hand side of the screen to move through the test. Under no circumstances use the Back or Forward buttons on your browser or the backspace button on your keyboard. If you finish ahead of time, you may go back over your answers if you wish or click the FINISH button. During the test you may not use any other software or programs on the computer or navigate away from the testing system website. If you are administering TAR45, say: You will have 20 minutes to complete the test. Please click on BEGIN TEST. If you are administering TAR60, say: You will have 25 minutes to complete the test. Please click on BEGIN TEST. If administering TAR45, wait approximately 20 minutes. If administering TAR60, wait approximately 25 minutes. The system is timing each individual but you should also keep an independent watch on the time. NOTE: If a candidate goes over time but does not click NEXT, PREVIOUS or FINISH on the item they are currently working on, it may appear that the system has not timed them out. When they eventually click NEXT, PREVIOUS or FINISH, the system will time the session out and close the test. In such a case when you are sure the time has gone past the time limit, ask the candidate to click one of the buttons. When all the candidates have been timed out, say: The time limit is up. Please click on the FINISH button to submit your responses. Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

29 Technical Information Users of ACER TAR should note that the study samples reported in this manual are not stratified samples of the Australian population. However, percentiles, stanines and scale scores have been derived from these reference groups to provide a point of comparison with an individual candidate s performance on the test. Demographic details of the samples used in each study are provided. Summary of research studies Study 1 was conducted on 151 adult volunteers from university, TAFE campuses, Centrelink and Job Network Providers using mainly TAR45 paper and pencil tests. Key descriptive, reliability and validity findings are given in the table below. Table 7 Key statistical findings in Study 1 Statistic Mean raw score Value out of 45 items Std. deviation 6.58 Internal reliability Correlation with SPM Demographic variable(s) that showed a significant difference in test scores 0.87 (Good) 0.80 (High) Age groups Study 2 was conducted on 295 adults using mainly TAR60 paper and pencil tests. Key descriptive, reliability and validity findings are given in the table below. Table 8 Key statistical findings in Study 2 Statistic Mean raw score Value out of 60 items Std. deviation 6.71 Internal reliability Correlation with APM Demographic variable(s) that showed a significant difference in test scores 0.84 (Good) 0.76 (High) Highest level of education completed Data for Study 3 was obtained from a total of 329 adult jobseekers who had completed TAR45 online. Key descriptive and reliability findings are given in the table below. Table 9 Key statistical findings in Study 3 Statistic Mean raw score Value 36.6 out of 45 items Std. deviation 7.29 Internal reliability

30 Te c h n i c a l I n f o r m a t i o n Study 1 ACER TAR Short Form (TAR45) Study design Study sample The data reported in this study were collected from a sample of 151 adults in Australia. The study participants were adult volunteers, including 30 applicants from a graduate recruitment program. The candidates were recruited from university and TAFE campuses, Centrelink and Job Network Provider premises, community organisations and other sources. The participants were asked to provide information on age, gender, occupation and highest level of education. Tables 10 to 15 describe the sample by the demographic data collected. Table 10 Demographics Gender Gender Number of participants Female 87 Male 51 No data 13 Total 151 Table 11 Demographics Highest level of education completed Highest level of education completed Number of participants Postgraduate degree 22 Undergraduate degree 35 TAFE diploma 13 Year Year 11 8 Year 10 1 No data 4 Total

31 A C E R Te s t o f A b s t r a c t R e a s o n i n g Table 12 Demographics Occupational classification Occupational classification Number of participants Student 77 Unskilled 2 Skilled/Trade 2 Office 22 Retail 5 Artist 3 Professional 26 No data/unemployed 14 Total 151 Table 13 Demographics Age group Age group Number of participants Under 20 years years years years years or over 16 No data 13 Total 151 Table 14 Demographics Language spoken at home Language spoken at home Number of participants English 78 Other 40 No data 33 Total 151 Testing procedure The tests were administered using standardised instructions provided in the test manual. Each study participant completed both ACER TAR45 and Raven s Standard Progressive Matrices in a single session. The participants were given 20 minutes to complete each test. Approximately half the participants completed ACER TAR45 before the SPM, and the other half in reverse order. Some participants managed to complete some of the extra 15 items that comprise TAR60. 24

32 Te c h n i c a l I n f o r m a t i o n Table 15 Demographics Order of tests Test order Number of participants ACER TAR45 SPM 70 SPM ACER TAR45 78 No data 3 Total 151 Item analysis QUEST analysis provides measures of general acceptability, content validity, contribution to internal consistency and sensitivity of individual items in a test. This information can be used to determine the usefulness and/or suitability of items for inclusion in a test. QUEST analysis was performed on all 60 TAR items and the results were tabulated (See Appendix A). This provided the: (a) number of correct responses obtained on each item and the number of persons who attempted the question, from which the difficulty estimates (measured in logits) for each item could be calculated; (b) the margin of error associated with the difficulty level of each item; (c) the infit mean squares, which are an indication of how well the item discriminates between poor and high performers. Infit scores of between 0.6 and 1.4 are generally considered acceptable. Infit scores less than 0.6 are considered problematic as they are too sensitive in discriminating performance while infit scores more than 1.4 have poor discriminating power. Because not all participants carried on after item 45, the item analysis was done first on all 60 TAR items and then on the first 45 TAR items. The item analysis done on all the 60 TAR items shows a high level of missing responses after item 45. As seen in the difficulty estimates, the difficulty levels of items 46 to 60 are generally much higher than the first 45 items. As seen from the results, the items cover a range of difficulties with 9 of the 60 items (items 1, 3, 8, 12, 15, 37, 38, 40, 42) showing some deviation from acceptable discrimination levels, as measured by the infit mean squares. Further research and analysis would need to be conducted to determine if these items should be excluded from ACER TAR. A further QUEST analysis was done on TAR45 that provided the following data: (a) facility, which is the percentage of individuals who got the item correct; (b) point biserial, which is the correlation between scores on items and overall test scores. This is an indication of how related the item is to the overall test. A positive number is considered appropriate; (c) difficulty estimates, measured in logits, which provide an indication of how difficult each item is; and (d) infit mean squares, which are an indication of how well the item discriminates between poor and high performers. Again, a value of between 0.6 and 1.4 is considered acceptable. As seen in Appendix B, the items cover an appropriate range of difficulty levels and all items are positively correlated with the overall test scores. All items are also within acceptable levels of discrimination. The difference in results of item analyses on TAR45 versus all 60 items may be due to the high number of missing scores from items 46 to 60. Reliability QUEST analysis on the data in this study indicated that the internal consistency of the TAR45 items was This compares to 0.90 for the full 60 items of the SPM for the study sample. These are considered strong measures of internal consistency, indicating that the items are highly inter-correlated and measure the same general construct. The slightly smaller value of the consistency of TAR45 items is likely to be related to the inclusion of only 45 items in the analysis compared to 60 items for the SPM. 25

33 A C E R Te s t o f A b s t r a c t R e a s o n i n g TAR45 subsets TAR45 has five sets of items each with its own instruction. Table 16 TAR45 set descriptions Set Set 1 (items 1 7) Set 2 (items 8 16) Set 3 (items 17 26) Set 4 (items 27 36) Set 5 (items 37 45) Instruction text The three shapes in the top row are alike in some way. Which shape in the bottom row is most like them? In the top row there are five squares with shapes arranged in order. One shape is missing. Which shape in the bottom row best completes the pattern? If the five shapes in the row are placed in logical order, which shape comes in the middle? Four of the five shapes are alike in some way. Which one is different? There is one missing shape in the pattern on the top. Which shape in the bottom row best completes the pattern? Correlations Set correlations An analysis of the internal correlations between the five sets in TAR45 was carried out and the results are shown in Table 17. Table 17 TAR45 internal set correlations Set 1 (items 1 7) 1 Set 2 (items 8 16) Set 1 Set 2 Set 3 Set 4 Set 3 (items 17 26) Set 4 (items 27 36) Set 5 (items 37 45) The low correlations between the sets indicate that each set of items may be measuring slightly different skills. Table 18 TAR45 set correlations with total score Correlations with total score on first 45 items Set 1 (items 1 7) 0.53 Set 2 (items 8 16) 0.69 Set 3 (items 17 26) 0.66 Set 4 (items 27 36) 0.79 Set 5 (items 37 45) 0.76 Each set, however, has correlations of considerable size with the total TAR45 score. This shows that the various items are measuring different skills that contribute to a measure of general ability. 26

34 Te c h n i c a l I n f o r m a t i o n Set reliability Inter-item correlations were performed on each set of items, which produced the results shown in Table 19. Table 19 TAR45 set reliabilities Number of items Reliabilities Set 1 (items 1 7) Set 2 (items 8 16) Set 3 (items 17 26) Set 4 (items 27 36) Set 5 (items 37 45) As seen in the table, set 1 has poor inter-item reliability. This lower correlation in set 1 could be due to the fact that it has fewer items than the other sets. However, inter-item correlations show that items 2 and 3 do not correlate well with other items in the same set. The deleted item statistics show that if these two items are removed, the reliability will be higher. Removal of item 2 raises the reliability from 0.34 to 0.39, and removal of item 3 raises the reliability from 0.34 to It was decided not to remove these items from the final test form until further data had been collected to substantiate these conclusions. Validity Correlations with Raven s Standard Progressive Matrices (SPM) The correlation between the SPM raw scores and the TAR45 raw scores is Correlation between the logits is This value shows that there is little difference between the two tests and they basically measure the same abilities. Removing 18 students brings most of the points within and near the 95% confidence belts. The shift constant, which is a measure of the difference between TAR45 and SPM, is Figure 5 95% confidence belt for equating between TAR45 logits and SPM logits A conversion table between SPM raw scores and TAR45 raw scores was calculated in the following way: (a) Corresponding to each SPM raw score, there is an SPM logit measure of the ability, which can be derived from the QUEST analysis. 27

35 A C E R Te s t o f A b s t r a c t R e a s o n i n g (b) Corresponding to each TAR45 raw score, there is also a TAR45 logit measure of the ability, which can also be obtained from the QUEST analysis. (c) Through equating, the TAR45 logit is put on the same scale as the SPM logit scale. This is done by adding the shift constant of 0.64 to each of the TAR45 logits. (d) The adjusted TAR45 logit is equated to the nearest SPM logit. So, a TAR45 raw score can provide a TAR45 logit measure of ability, which after adjusting for the shift constant is equated to the nearest SPM logit, from which an SPM raw score can be determined. A detailed conversion table for TAR45 raw scores to SPM raw scores is provided in Appendix C. Scale scores TAR45 has been calibrated using the Rasch measurement model (Fisher and Molenaar, 1995). The average difficulty was then set at 65 with a standard deviation of 15. The Rasch model gives an ability estimate for each person in terms of logit. The logits are then transformed into scaled scores with a range from 0 to 100 and a standard deviation of 15. The mean of the scaled scores is chosen so that a student answering all TAR45 items correctly will be given a scaled score of 100. The mean scale score obtained from this sample of 151 students is 65. Demographic variables Means and standard deviations were calculated for the study participants by five background variables: gender, level of education, occupational classification, language background, age and order of test. An analysis of variance (ANOVA) was also performed for each background variable to find out if they had significantly different scores. In each case candidates providing no data were counted as one group. A significance value in the ANOVA tables of less than or equal to 0.05 is indicative of a statistically significant difference in the performance of the different groups. Gender Results of males and females were compared and analysed. The results in Tables 20 and 21 show that there was no significant difference in the performance of males and females on TAR45. Table 20 TAR45 mean and standard deviation by gender Missing scores Female scores Male scores Group total Count Mean Maximum Minimum Std. deviation Table 21 ANOVA TAR45 scores on gender Sum of squares df Mean square F Significance value Between groups Within groups Total

36 Te c h n i c a l I n f o r m a t i o n Education level Individual results were grouped by education level and analysed. The results in Tables 22 and 23 show that there was no significant difference in the scores of individuals in different educational levels even though there was some difference in the mean scores of Year 10 students and TAFE diploma students. Table 22 TAR45 mean and standard deviation by education level Postgrad degree Undergrad degree TAFE diploma Year 12 Year 11 Year 10 No data Group total Count Mean Maximum Minimum Std. deviation Table 23 ANOVA TAR45 scores on educational level Sum of squares df Mean square F Significance value Between groups Within groups Total Occupational classification Individual results were grouped by occupational classification and analysed. The results in Tables 24 and 25 show that there was no significant difference in the scores of individuals in different occupational groups. However, the mean score of those who are unemployed or have provided no data was considerably lower than that of the other occupational groups. Table 24 TAR45 mean and standard deviation by occupational classification Student Unskilled Trade Office Retail Artist Professional No data/ Unemployed Group total Count Mean Maximum Minimum Std. deviation

37 A C E R Te s t o f A b s t r a c t R e a s o n i n g Table 25 ANOVA TAR45 scores on occupational classification Sum of squares df Mean square F Significance value Between groups Within groups Total Language background Individual results were grouped by whether they were from an English-speaking background and analysed. The results in Tables 26 and 27 show that there was no significant difference in the scores of individuals from different language backgrounds. Table 26 TAR45 mean and standard deviation by language English Other no data Group total Count Mean Maximum Minimum Std. deviation Table 27 ANOVA TAR45 scores on language Sum of squares df Mean squares F Signigicance value Between groups Within groups Total Age groups Individual results were grouped by age and analysed. The results in Tables 28 and 29 show that there was a significant difference in the scores of individuals of different ages. Specifically, mean scores on TAR45 increased until age 29 and then started to decline in those aged 30 and above. Table 28 TAR45 mean and standard deviation by age groups <20 20 to to to or over Missing Group total Count Mean Minimum Maximum Std. deviation

38 Te c h n i c a l I n f o r m a t i o n Table 29 ANOVA TAR45 scores on age groups Sum of squares df Mean square F Significance value Between groups Within groups Total Test order Two groups of individuals were given the SPM and TAR45 in different order. One group was given the SPM first and then TAR45. The other group was given the two tests in the reverse order. This was to negate any possible order or practise effect. The results in Tables 30 and 31 show that there was no significant difference in the scores of the two groups. Table 30 TAR45 mean and standard deviation by test order SPM/TAR45 TAR45/SPM No data Group total Count Mean Maximum Minimum Std. deviation Table 31 ANOVA TAR45 scores on test order Sum of squares df Mean square F Significance value Between groups Within groups Total Conclusion In these analyses of variance, the only variable to show a degree of significance was age groups. No significant differences were found for gender, education level, occupation, language background or test order. 31

39 A C E R Te s t o f A b s t r a c t R e a s o n i n g Study 2 ACER TAR Long Form (TAR60) Study design Study sample The data reported in this study were collected from a sample of 295 adults in Australia. The participants were asked to provide information on gender, education level, study type and language background. Participants in the study consisted of 212 undergraduates (72%), 28 postgraduates (9%) and 55 participants (19%) who were not students or who did not provide information. 225 participants (76%) had completed Year participants (16%) possessed undergraduate degrees. 8 (about 3%) held postgraduate degrees. Only 5 (2%) in the sample had not completed Year 12. The sample is fairly evenly represented in gender. There were 139 (47%) females and 154 (52%) males. Only two participants (1%) did not provide their gender information. Tables describe the sample by the demographic data collected. Table 32 Demographics Study types Study type Number of participants Undergraduate/humanities, law, education 55 Undergraduate/science, IT, engineering 88 Undergraduate/commerce, finance, accounting 22 Undergraduate/medicine 15 Undergraduate/double degree 26 Undergraduate/not stated 6 Postgraduate/commerce 6 Postgraduate/humanities 6 Postgraduate/science 16 Not stated/not a student 55 Total 295 Table 33 Demographics Highest level of education completed Highest level of education completed Number of participants Postgraduate degree 8 Undergraduate degree 48 TAFE diploma 8 Year Year 11 3 Year 10 2 No data 1 Total

40 Te c h n i c a l I n f o r m a t i o n Table 34 Demographics Gender Gender Number of participants Female 139 Male 154 No data 2 Total 295 Table 35 Demographics Language spoken at home Language spoken at home Number of participants English 165 Other 128 No data 2 Total 295 Testing procedure The tests were administered using standardised instructions provided in the test manuals. Each study participant completed both TAR60 and the Raven s Advanced Progressive Matrices (APM) in a single session. The participants were given 25 minutes to complete each test. Approximately half the participants completed the TAR60 before the APM, and the other half in reverse order. This was to check for possible order or practise effects. Table 36 Demographics Order of tests Test order Number of participants ACER TAR60 APM 152 APM ACER TAR Total 295 Item analysis Because of the high number of missing responses after item 45 on the TAR60, the item analysis was done first on all 60 TAR items and then on the first 45 items. The QUEST Analysis provided the following information (see Appendix D): (a) number of correct responses obtained on each item and the number of persons who attempted the question, from which the difficulty estimates (measured in logits) for each item could be calculated; (b) the margin of error associated with the difficulty level of each item; (c) the infit mean squares, which are an indication of how well the item discriminates between poor and high performers. Infit scores of between 0.6 and 1.4 are generally considered acceptable. Infit scores less than 0.7 are considered problematic as they are too sensitive in discriminating performance while infit scores more than 1.3 have poor discriminating power. As seen from the results in Appendix D, the items cover a range of difficulties with 3 of the 60 items (items 8, 16 and 37) showing any deviation from acceptable discrimination levels, as measured by the infit mean squares. The difficulty levels of items 46 to 60 are generally much higher than the first 45 items. 33

41 A C E R Te s t o f A b s t r a c t R e a s o n i n g A further QUEST analysis was done on the first 45 TAR items, which provided the following information (See Appendix E): (a) facility, which is the percentage of individuals who got the item correct; (b) point biserial, which is the correlation between scores on items and overall test scores. This is an indication of how related the item is to the overall test and a positive number is considered appropriate; (c) difficulty estimates, measured in logits, which provide an indication of how difficult each item is; and (d) infit mean squares, which are an indication of how well the item discriminates between poor and high performers. Again, a value of between 0.6 and 1.4 is considered acceptable. Reliability The internal reliability of TAR60 items was For TAR45, the internal reliability is The internal reliability of the APM, which has 36 items and which is also administered to this sample of students, has a reliability of These are all acceptable levels of internal consistency, which means that the items in each test inter-correlate and generally measure the same underlying dimension. TAR60 subsets TAR60 has seven sets of items, each with its own instruction. Table 37 TAR60 set descriptions Set Set 1 (items 1 7) Set 2 (items 8 16) Set 3 (items 17 26) Set 4 (items 27 36) Set 5 (items 37 45) Set 6 (items 46 55) Set 7 (items 56 60) Instruction text The three shapes in the top row are alike in some way. Which shape in the bottom row is most like them? In the top row there are five squares with shapes arranged in order. One shape is missing. Which shape in the bottom row best completes the pattern? If the five shapes in the row are placed in logical order, which shape comes in the middle? Four of the five shapes are alike in some way. Which one is different? There is one missing shape in the pattern on the top. Which shape in the bottom row best completes the pattern? The four shapes in the top row are in order. Which shape in the bottom row logically follows the fourth shape in the top row? There is one missing shape in the pattern at the top. Which shape in the bottom row best completes the pattern? Correlations Set correlations An analysis of the internal correlations between the five sets in TAR45 was carried out and the results are shown in the tables below. 34

42 Te c h n i c a l I n f o r m a t i o n Table 38 TAR60 internal set correlations Set 1 Set 2 Set 3 Set 4 Set 5 Set 6 Set 1 (items 1 7) 1 Set 2 (items 8 16) Set 3 (items 17 26) Set 4 (items 27 36) Set 5 (items 37 45) Set 6 (items 46 55) Set 7 (items 56 60) Table 39 TAR60 set correlations with total score Correlations with total score on first 45 items Correlations with total score on all 60 items Set 1 (items 1 7) Set 2 (items 8 16) Set 3 (items 17 26) Set 4 (items 27 36) Set 5 (items 37 45) Set 6 (items 46 55) 0.81 Set 7 (items 56 60) 0.62 The low correlations between the sets seen in Table 38 indicate that each set of items may be measuring separate components. Each set, however, has correlations of considerable size with the total TAR60 score. Set reliability Table 40 TAR60 set reliabilities Number of items Reliabilities Set 1 (items 1 7) Set 2 (items 8 16) Set 3 (items 17 26) Set 4 (items 27 36) Set 5 (items 37 45) Set 6 (items 46 55) Set 7 (items 56 60)

43 A C E R Te s t o f A b s t r a c t R e a s o n i n g Reliability depends on the length of the test and longer tests usually have higher reliability. This is reflected to a certain degree in Table 40. Set 6, with 10 items, has a reliability of 0.80 and set 1, with 7 items, has a reliability of only Set 2 has a low reliability based on data from this sample. Additional item analysis showed that item 15 has a low point biserial coefficient and item 16 has a low facility. Removing these two items from the set, though shortening its length, raises the reliability to Apart from set 2, the internal reliability for each set is acceptable. If each set is allowed to have 20 items, the reliability in each set, with the exception of set 2, will reach 0.7 using the Spearman-Brown formula. Validity Correlations with Raven s Advanced Progressive Matrices (APM) Correlations between TAR60 and APM were conducted to demonstrate construct validity, which shows that the two tests measure the same underlying ability. The correlation between APM raw scores and TAR60 raw scores is Table 41 TAR60 set correlations with APM total score Correlations with APM total score Set 1 (items 1 7) 0.41 Set 2 (items 8 16) 0.23 Set 3 (items 17 26) 0.32 Set 4 (items 27 36) 0.27 Set 5 (items 37 45) 0.35 Set 6 (items 46 55) 0.53 Set 7 (items 56 60) 0.40 Set correlations Set 6 of TAR60, containing items 46 55, is the set with the highest correlation with the APM total scores. This explains why the correlation between TAR60 scores and APM scores is higher if the last 15 items are included. Set 2 has the lowest correlation with APM total. Table 42 Correlations of TAR60 items with APM total score Correlations with APM total scores (Set 1) Correlations with APM total scores (Set 2) 1st item nd item rd item th item th item th item th item th item th item

44 Te c h n i c a l I n f o r m a t i o n As an illustration, correlations between item scores in set 1 and set 2 in ACER TAR and the APM total are given in Table 42. These in fact are the point biserial of the items but use APM total scores in the correlation instead of the ACER TAR total. If ACER TAR and APM are measuring the same construct, the point biserial should be similar whether using ACER TAR total or APM total. It is seen that items in set 1 are satisfactory but there are some items in set 2 with negative correlations or correlations very close to zero. Calculating the scale scores TAR60 has been calibrated using the Rasch measurement model (Fisher and Molenaar, 1995). The Rasch model gives an ability estimate for each person in terms of logit. The logits are then transformed into scaled scores with a range from 0 to 100 and a standard deviation of 15. The mean of the scaled scores is chosen so that a student answering all TAR60 items correctly will be given a scaled score of 100. The mean scale score obtained from this sample of 295 students is 65. How well the data fit the Rasch model is indicated by the infit mean squares. Infit scores of between 0.6 and 1.4 are generally considered acceptable. Infit scores less than 0.7 are considered problematic as they are too sensitive in discriminating performance while infit scores more than 1.3 have poor discriminating power. Equating TAR60 to APM The APM logits and the TAR60 logits of the same students are plotted in a graph after making adjustments for the scale. Not all of the 295 students lie within the 95% confidence belts. This means that after adjustment of the scale, the logit measures from TAR60 and those from APM are not likely to come from the same person. In the process of test equating, these persons will be removed in order to find an appropriate constant for adjustment of the scale. For TAR60, the shift constant found is A conversion table between APM raw scores and TAR60 raw scores was calculated in the following way: (a) Corresponding to each APM raw score, there is an APM logit measure of the ability, which can be derived from the QUEST analysis. (b) Corresponding to each TAR60 raw score, there is a TAR60 logit measure of the ability, which can also be obtained from the QUEST analysis. (c) Through equating, the TAR60 logit is put on the same scale as the APM logit scale. This is done by adding the shift constant of to each of the TAR60 logits. (d) The adjusted TAR60 logit is equated to the nearest APM logit. Figure 6 95% confidence belt for equating between TAR60 logits and APM logits 37

45 A C E R Te s t o f A b s t r a c t R e a s o n i n g So, a TAR60 raw score yields a TAR60 logit measure of ability, which is equated to the nearest APM logit. After adjusting for the shift constant this is then used to determine an APM raw score (See Appendix F). Background variables Means and standard deviations were calculated for the study participants by five background variables: gender, level of education, language background, study type and order of test. An analysis of variance (ANOVA) was also performed for each background variable. Gender Results of males and females were compared and analysed. The results in Tables 43 and 44 show that there was no significant difference in the performance of males and females on TAR60. Table 43 TAR60 mean and standard deviation by gender Male scores Female scores Missing scores Group total Valid N Mean Minimum Maximum Std. deviation Table 44 ANOVA TAR60 scores on gender Sum of squares df Mean square F Significance value Between groups Within groups Total Education level Individual results were grouped by education level and analysed. The results in Tables 45 and 46 show that there was no significant difference in the scores of individuals with different educational levels, although Year 12, undergraduate and postgraduate students scored higher than Years 10 and 11 students. Table 45 TAR60 mean and standard deviation by education level Postgrad degree Undergrad degree TAFE diploma Year 12 Year 11 Year 10 Missing Group total Valid N Mean Minimum Maximum Std. deviation

46 Te c h n i c a l I n f o r m a t i o n Table 46 ANOVA TAR60 scores on educational level Sum of squares df Mean square F Significance value Between groups Within groups Total Language background Individuals were grouped according to whether or not they were from an English-speaking background and their results were analysed. The results in Tables 47 and 48 show that there was no significant difference in the scores of individuals from different language backgrounds, indicating that TAR60 is a relatively language-free measure of general ability. Table 47 TAR60 mean and standard deviation by language English Other Missing Group total Valid N Mean Minimum Maximum Std. deviation Table 48 ANOVA TAR60 scores on language Sum of squares df Mean square F Significance value Between groups Within groups Total Study type Individual results were grouped by study type and analysed. The results in Tables 49 and 50 show that there was no significant difference in the scores of individuals in different types of study. Table 49 TAR60 mean and standard deviation by study types Valid N Mean Minimum Maximum Std. deviation Key: 1. Undergraduate humanities, law or education degree; 2. Undergraduate science, IT or engineering degree; 3. Undergraduate commerce, finance or accounting degree; 4. Undergraduate medicine degree; 5. Undergraduate double degree; 6. Undergraduate degree discipline not stated; 7. Postgraduate commerce degree; 8. Postgraduate humanities degree; 9. Postgraduate science degree; 10. Not stated / not a student; 11. Group total. 39

47 A C E R Te s t o f A b s t r a c t R e a s o n i n g Table 50 ANOVA TAR60 scores on study types Sum of squares df Mean square F Significance value Between groups Within groups Total Test order Two groups of individuals were given the APM and TAR60 in different order. One group was given the APM before TAR60. The other group was given the two tests in the reverse order. This was to negate the possible order or practise effect. The results in Tables 51 and 52 show that there was no significant difference in the scores of the two groups. Table 51 TAR60 mean and standard deviation by test order TAR60/APM APM/TAR60 Group total Valid N Mean Minimum Maximum Standard deviation Table 52 ANOVA TAR60 scores on test order Sum of squares df Mean square F Significance value Between groups Within groups Total ACER TAR Short Form (TAR45) The following is analysis on data from the first 45 items of TAR60 completed by the sample in Study 2. Because there were fewer missing responses on the first 45 items, this provides more accurate data. Background variables Gender Results of males and females were compared and analysed. The results in Tables 53 and 54 show that there was no significant difference in the performance of males and females. 40

48 Te c h n i c a l I n f o r m a t i o n Table 53 TAR45 mean and standard deviation by gender Male scores Female scores Missing Group total Valid N Mean Minimum Maximum Std. deviation Table 54 ANOVA TAR45 scores on gender Sum of squares df Mean square F Significance value Between groups Within groups Total Education level Individual results were grouped by education level and analysed. The results in Tables 55 and 56 show that there is no significant difference in the scores of individuals in different educational levels, however, there is some difference in the mean scores of Year 10 students compared to undergraduate and postgraduate students. Table 55 TAR45 mean and standard deviation by education level Postgrad degree Undergrad degree TAFE diploma Year 12 Year 11 Year 10 Missing Group total Valid N Mean Minimum Maximum Std. deviation Table 56 ANOVA TAR45 scores on educational level Sum of squares df Mean square F Significance value Between groups Within groups Total

49 A C E R Te s t o f A b s t r a c t R e a s o n i n g Language background Individual results were grouped by language background and analysed. The results in Tables 57 and 58 show that there is no significant difference in the scores of individuals from different language backgrounds. Table 57 TAR45 mean and standard deviation by language English Other Missing Group total Valid N Mean Minimum Maximum Std. deviation Table 58 ANOVA TAR45 scores on language Sum of squares df Mean square F Significance value Between groups Within groups Total Study type Individual results were grouped by study type and analysed. The results in Tables 59 and 60 show that there is no significant difference in the scores of individuals in different types of study. Table 59 TAR45 mean and standard deviation by study types Valid N Mean Minimum Maximum Std. deviation Key: 1. Undergraduate humanities, law or education degree; 2. Undergraduate science, IT or engineering degree; 3. Undergraduate commerce, finance or accounting degree; 4. Undergraduate medicine degree; 5. Undergraduate double degree; 6. Undergraduate degree discipline not stated; 7. Postgraduate commerce degree; 8. Postgraduate humanities degree; 9. Postgraduate science degree; 10. Not stated / not a student; 11. Group total. 42

50 Te c h n i c a l I n f o r m a t i o n Table 60 ANOVA TAR45 scores on study types Sum of squares df Mean square F Significance value Between groups Within groups Total Test order Two groups of individuals were given the APM and TAR60 in different order. One group was given the APM before TAR60. The other group was given the tests in the reverse order. This was to negate possible order or practise effects. The results in Tables 61 and 62 show that there was no significant difference in the scores of the two groups. Table 61 TAR45 mean and standard deviation by test order TAR45/APM APM/TAR45 Group total Valid N Mean Minimum Maximum Std. deviation Table 62 ANOVA TAR45 scores on test order Sum of squares df Mean square F Significance value Between groups Within groups Total Conclusion In these analyses of variance no significant differences were found for gender, language, study type and order of test. However, a significant difference was found between groups with different levels of completed education on the TAR60 and TAR45. 43

51 A C E R Te s t o f A b s t r a c t R e a s o n i n g Study 3 ACER TAR Short Form (TAR45) 2006 Study design This study was conducted using data from the completion of TAR45 online from January 2005 to April The individuals included in this study were recruitment candidates or applicants for various job positions. There were a total of 329 individuals in this reference group. Percentiles and stanines derived from this reference group will be available in all automated reports. The number of males and females in the reference group is provided in the table below. Table 63 Number of males and females in the reference group Gender Number of participants Males 203 Females 126 Total 329 The descriptive statistics for TAR45 are provided in Table 64. The mean raw score on TAR45 is out of 45 questions for this group. The raw scores tend to be skewed towards the higher end for this group. Table 64 Descriptive statistics for TAR45 Value Valid N 326 Missing 3 Mean Std. deviation 6.36 Variance Range 32 Minimum 13 Maximum 45 Test score at 25th percentile 34 Test score at 50th percentile 39 Test score at 75th percentile 41 Reliability The reliabilities of the subscales comprising TAR45 are given below. The internal consistency of the items contained in the subsets is relatively high, ranging from 0.52 to This means that all the items in the subsets are measuring the same construct. The internal reliability of TAR45 was 0.904, which is very high. 44

52 Te c h n i c a l I n f o r m a t i o n Table 65 Reliabilities of TAR45 subscales TAR45 Number of items Reliabilities Set 1 (items 1 7) Set 2 (items 8 16) Set 3 (items 17 26) Set 4 (items 27 36) Set 5 (items 37 45) Table 66 gives the correlations between the various item subsets in TAR45. There are low to moderate correlations between the various subsets, which show that the constructs measured by each subset are related to some extent. Table 66 Correlations between subsets in TAR45 TAR45 Set 1 Set 2 Set 3 Set 4 Set 5 Set 1 (items 1 7) 1 Set 2 (items 8 16) Set 3 (items 17 26) Set 4 (items 27 36) Set 5 (items 37 45) Gender differences Data from this study was split into male and female scores and compared. Table 67 TAR45 scores for males and females Female raw score Male raw score Mean Maximum Minimum Range Std. deviation Count No significant difference between males and females was seen, based on the results of an independent samples t test, which checks for statistically significant differences between two or more groups. This shows that there is no difference in the performance of males and females on TAR45. 45

53 A C E R Te s t o f A b s t r a c t R e a s o n i n g Table 68 Statistics from independent sample t tests Test t statistics df Significance Test of Abstract Reasoning (TAR45) Conclusion TAR45 is a reliable, valid instrument that correlates well with Raven s SPM as a measure of general ability. 46

54 References Anastasi, A. 1976, Psychological Testing, 4th edn, Macmillan, New York. Australian Council for Educational Research 2000, Aptitude Profile Test Series (APTS) Occupational, ACER, Camberwell. Australian Council for Educational Research 2005, Middle Years Ability Tests (MYAT), ACER, Camberwell. Cronbach, J. L. 1970, Essentials of Psychological Testing, 3rd edn, Harper & Row, New York. Fisher, G. H. and Molenaar, I. W. 1995, Rasch Models: Foundations, Recent Developments and Applications, Springer-Verlag, New York. Holland, J. L. 1997, Making Vocational Choices: A Theory of Vocational Personalities and Work Environments, 3rd edn, Psychological Assessment Resources, Florida. Kummerow, J. M. (ed.) 1991, New Directions in Career Planning and the Workplace: Practical Strategies for Counselors, Davies-Black, California. Nelson Bolles, R. 2006, What Color Is Your Parachute?, Ten Speed Press, California. Power, M. 2004, A Quick Guide to HR Testing, ACER Press, Camberwell. Raven, J. C. et. al. 1983, Raven s Advanced Progressive Matrices, OPP Limited, Oxford. Raven, J. C. et. al. 2000, Raven s Standard Progressive Matrices 2000 Edition, OPP Limited, Oxford. 47

55 Appendices APPENDIX A: QUEST item analysis for all 60 TAR items (Study 1) Item number Correct responses Number of people attempting item Difficulty estimates (in logits) Error Infit mean squares

56 A p p e n d i c e s Appendix A (cont.) Item number Correct responses Number of people attempting item Difficulty estimates (in logits) Error Infit mean squares

57 A p p e n d i c e s APPENDIX B: QUEST output for TAR45 (Study 1) Item number Facility Point biserial Difficulty estimates (in logits) Infit mean squares

58 A p p e n d i c e s Appendix B (cont.) Item number Facility Point biserial Difficulty estimates (in logits) Infit mean squares

59 A p p e n d i c e s APPENDIX C: Detailed conversion table between TAR45 and SPM (Study 1) TAR raw TAR45 scale score TAR logit (pre-equating) TAR logit (post equating: including shift constant) SPM logit SPM raw * Numbers in bold are obtained by linear interpolation 52

60 A p p e n d i c e s APPENDIX D: QUEST item analysis for all 60 TAR items (Study 2) Item number Correct responses Number of people attempting them Difficulty estimates (in logits) Error Infit mean squares

61 A p p e n d i c e s Appendix D (cont.) Item number Correct responses Number of people attempting them Difficulty estimates (in logits) Error Infit mean squares

62 A p p e n d i c e s APPENDIX E: QUEST output for TAR45 (Study 2) Item number Facility Point biserial Difficulty estimates (in logits) Infit mean squares

63 A p p e n d i c e s Appendix E (cont.) Item number Facility Point biserial Difficulty estimates (in logits) Infit mean squares

64 A p p e n d i c e s APPENDIX F: Detailed conversion table between TAR60 and APM (Study 2) TAR60 raw score TAR60 scale score Best predicted TAR60 logit Best predicted TAR60 logit in TAR60 scale APM logit APM raw score

65

66 The ACER Test of Abstract Reasoning (TAR) is a quick and easily administered test of abstract reasoning in adults. Abstract reasoning is linked to the ability to think strategically, analyse information, learn new information quickly, engage in creative problem-solving and make logical decisions. TAR is suitable for use as a selection or development tool for graduate, managerial and other positions that require critical thinking skills. The test is free from language, cultural and gender biases. TAR is available online and in paper-based formats, and is easy to administer and interpret. It provides immediate results using Australian norms and can be used in individual or group testing situations. TAR can be used for various courses and occupations, including: Managers and executives Business managers Professionals and specialists Para-professionals (e.g. technicians) Arts professionals Finance and data analysts Medical professionals Scientists TAR ISBN 10: ISBN 13: Australian Council for Educational Research