TAR ACER. Test of Abstract Reasoning MANUAL

Transcription

1 TEST OF ABSTRACT REASONING TEST OF ABSTRACT REASONING TEST OF ABSTRACT REASONING TEST OF ABSTRACT REASONING TEST OF ABSTRACT ACER Test of Abstract Reasoning MANUAL TAR

2 ACER Press

3 First published 2007 by ACER Press Australian Council for Educational Research Ltd 19 Prospect Hill Road, Camberwell, Victoria, 3124 Copyright 2007 Australian Council for Educational Research All rights reserved. Except under the conditions described in the Copyright Act 1968 of Australia and subsequent amendments, no part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the written permission of the publishers. Only material contained in the photocopy masters may be reproduced, and then only in quantities sufficient for the purchaser s own proper use and not for resale to, or use by, any other person or organisation. Edited by Elisa Webb Cover design by R.T.J. Klinkhamer Typeset by Cannon Typesetting, Melbourne Printed by Print Impressions National Library of Australia Cataloguing-in-Publication data: Australian Council for Educational Research ACER test of abstract reasoning manual. Bibliography. For secondary students. ISBN ISBN Reasoning (Psychology) Testing Handbooks, manuals, etc. I. Title Visit our website:

4 Contents List of tables...v List of figures...vi List of appendices...vi About ACER Test of Abstract Reasoning (ACER TAR)... 1 Background...1 Description of the tests...1 Typical uses...2 Occupational uses...2 Educational and vocational guidance uses...2 Test Options... 3 Short or long form...3 Short Form (TAR45)...3 Long Form (TAR60)...3 Paper or online administration...3 Paper...3 Online...3 Scoring and Reporting... 4 Paper administration...4 ACER Test Scoring Services...4 Online Response Entry System...4 Online administration...5 Automated reporting...5 ACER Online Testing System...5 Direct request...5 Interpretation of ACER TAR Reports... 7 Interpreting norm scores...7 Percentile ranks...7 Stanine scores...7 Rasch scores...8 Comparison with Raven s Standard Progressive Matrices and Advanced Progressive Matrices...8 Case Studies Example 1: Recruitment for a public sector department...11 Example 2: Call centre selection...11 Example 3: Promoting floor staff to store manager...11 Directions for Administration General information...13 General information for online testing...13 Minimum system requirements...14 Specific instructions...14 Instructions for paper administration...15 Instructions for online administration...19 Technical Information Summary of research studies...22 Study 1 ACER TAR Short Form (TAR45)...23 Study design...23 Study sample...23 Testing procedure...24 iii

5 C o n t e n t s Item analysis...25 Reliability...25 TAR45 subsets...26 Correlations...26 Set correlations...26 Set reliability...27 Validity...27 Correlations with Raven s Standard Progressive Matrices...27 Scale scores...28 Demographic variables...28 Gender...28 Education level...29 Occupational classification...29 Language background...30 Age groups...30 Test order...31 Conclusion...31 Study 2 ACER TAR Long Form (TAR60)...32 Study design...32 Study sample...32 Testing procedure...33 Item analysis...33 Reliability...34 TAR60 subsets...34 Correlations...34 Set correlations...34 Set reliability...35 Validity...36 Correlations with Raven s Advanced Progressive Matrices...36 Set correlations...36 Calculating the scale scores...37 Equating TAR60 to APM...37 Background variables...38 Gender...38 Education level...38 Language background...39 Study type...39 Test order...40 ACER TAR Short Form (TAR45)...40 Background variables...40 Gender...40 Education level...41 Language background...42 Study type...42 Test order...43 Conclusion...43 Study 3 ACER TAR Short Form (TAR45) Study design...44 Reliability...44 Gender differences...45 Conclusion...46 References...47 Appendices...48 iv

6 List of Tables Table 1 Source of items in ACER TAR...1 Table 2 Example occupation groups...2 Table 3 Examples of appropriate forms for selected jobs...3 Table 4 Stanine groups and descriptors...7 Table 5 Simple conversion table between ACER TAR45 and Raven s SPM...9 Table 6 Simple conversion table between ACER TAR60 and Raven s APM...10 Table 7 Key statistical findings in Study Table 8 Key statistical findings in Study Table 9 Key statistical findings in Study Table 10 Demographics Gender...23 Table 11 Demographics Highest level of education completed...23 Table 12 Demographics Occupational classification...24 Table 13 Demographics Age group...24 Table 14 Demographics Language spoken at home...24 Table 15 Demographics Order of tests...25 Table 16 TAR45 set descriptions...26 Table 17 TAR45 internal set correlations...26 Table 18 TAR45 set correlations with total score...26 Table 19 TAR45 set reliabilities...27 Table 20 TAR45 mean and standard deviation by gender...28 Table 21 ANOVA TAR45 scores on gender...28 Table 22 TAR45 mean and standard deviation by education level...29 Table 23 ANOVA TAR45 scores on educational level...29 Table 24 TAR45 mean and standard deviation by occupational classification...29 Table 25 ANOVA TAR45 scores on occupational classification...30 Table 26 TAR45 mean and standard deviation by language...30 Table 27 ANOVA TAR45 scores on language...30 Table 28 TAR45 mean and standard deviation by age groups...30 Table 29 ANOVA TAR45 scores on age groups...31 Table 30 TAR45 mean and standard deviation by test order...31 Table 31 ANOVA TAR45 scores on test order...31 Table 32 Demographics Study types...32 Table 33 Demographics Highest level of education completed...32 Table 34 Demographics Gender...33 Table 35 Demographics Language spoken at home...33 Table 36 Demographics Order of tests...33 Table 37 TAR60 set descriptions...34 Table 38 TAR60 internal set correlations...35 Table 39 TAR60 set correlations with total score...35 Table 40 TAR60 set reliabilities...35 Table 41 TAR60 set correlations with APM total score...36 Table 42 Correlations of TAR60 items with APM total score...36 Table 43 TAR60 mean and standard deviation by gender...38 Table 44 ANOVA TAR60 scores on gender...38 Table 45 TAR60 mean and standard deviation by education level...38 Table 46 ANOVA TAR60 scores on educational level...39 Table 47 TAR60 mean and standard deviation by language...39 Table 48 ANOVA TAR60 scores on language...39 Table 49 TAR60 mean and standard deviation by study types...39 Table 50 ANOVA TAR60 scores on study types...40 v

7 L i s t o f Ta b l e s Table 51 Table 52 Table 53 Table 54 Table 55 Table 56 Table 57 Table 58 Table 59 Table 60 Table 61 Table 62 Table 63 Table 64 Table 65 Table 66 Table 67 Table 68 TAR60 mean and standard deviation by test order...40 ANOVA TAR60 scores on test order...40 TAR45 mean and standard deviation by gender...41 ANOVA TAR45 scores on gender...41 TAR45 mean and standard deviation by education level...41 ANOVA TAR45 scores on education level...41 TAR45 mean and standard deviation by language...42 ANOVA TAR45 scores on language...42 TAR45 mean and standard deviation by study types...42 ANOVA TAR45 scores on study types...43 TAR45 mean and standard deviation by test order...43 ANOVA TAR45 scores on test order...43 Number of males and females in the reference group...44 Descriptive statistics for TAR Reliabilities of TAR45 subscales...45 Correlations between subsets in TAR TAR45 scores for males and females...45 Statistics from independent sample t tests...46 List of Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Example of a group report from ACER Test Scoring Services...4 Example of an automated report...5 Example of a group report from the Practitioner s site...6 Bell-shaped distribution % confidence belt for equating between TAR45 logits and SPM logits % confidence belt for equating between TAR60 logits and APM logits...37 List of Appendices Appendix A QUEST item analysis for all 60 TAR items (Study 1)...48 Appendix B QUEST output for TAR45 (Study 1)...50 Appendix C Detailed conversion table between TAR45 and SPM (Study 1)...52 Appendix D QUEST item analysis for all 60 TAR items (Study 2)...53 Appendix E QUEST output for TAR45 (Study 2)...55 Appendix F Detailed conversion table between TAR60 and APM (Study 2)...57 vi

8 About ACER Test of Abstract Reasoning (ACER TAR) Background ACER Test of Abstract Reasoning (ACER TAR) has been developed in response to demand for an abstract reasoning assessment with Australian adult data. The items from ACER TAR have been sourced from two existing, well-researched ACER tests of abstract reasoning: the Middle Years Ability Test (MYAT) and the Aptitude Profile Test Series (APTS). Items from these two existing tests have been combined to create the ACER TAR Long and Short forms: Table 1 Source of items in ACER TAR Test form ACER TAR Short Form (TAR45) ACER TAR Long Form (TAR60) Item source 45 abstract reasoning items from MYAT 45 abstract reasoning items from MYAT 15 abstract reasoning items from APTS Items were chosen to reflect an appropriate range of difficulty and task sub-type. The data samples detailed in this manual are based on: (a) a study conducted on 151 adults in Australia who completed both ACER TAR and the Raven s Standard Progressive Matrices (SPM); (b) a study conducted on 295 adults at an Australian university who completed both ACER TAR and the Raven s Advanced Progressive Matrices (APM); and (c) a study conducted on 329 adults seeking work who had completed ACER TAR online. Further research studies and analyses of ongoing data collection projects will be reported in future editions of this manual. Description of the tests ACER TAR is a test of abstract (or non-verbal) reasoning. The MYAT Manual defines abstract (non-verbal) reasoning as the extent to which [a person] can discover principles and rules, and apply them to solve problems using abstract visual patterns rather than numbers and words (p. 1). The APTS Manual defines abstract reasoning as the ability to discover the principle/s [that] determine the rule or rules that govern the progression of a pattern from one diagram to another in a series, or to identify the part which is missing from a diagram (p. 3). Tests of abstract reasoning such as the ACER TAR are nearly language-free and so may avoid problems associated with the use of verbal reasoning tests in some contexts where candidates written English skills are not well developed, such as where candidates are from non-english speaking backgrounds or have specific language learning difficulties. Statistical analyses detailed in this manual (see Technical information, p. 22) confirm that language background is not a significant source of variance in performance on the ACER TAR.

9 A C E R Te s t o f A b s t r a c t R e a s o n i n g Typical uses Occupational uses ACER TAR is typically used as a measure of general ability for selection into occupations which involve a moderate to high level of demand on reasoning ability and for other purposes where the ability to think clearly, analyse information, solve problems and make rational decisions are important. Examples of selection criteria linked to measures of abstract reasoning are: Planning and coordinating activities Creating innovative solutions to problems Making decisions in a rational manner Learning new material quickly Collecting and analysing information (Power, 2004, p. 24) Table 2 provides examples of occupational groups for which the ACER TAR can be used. Table 2 Example occupation groups Main occupation group Managers and administrators Professionals Para-professionals Tradespersons Artistic and creative Figures and computational Literary Medical Technical and engineering Scientific Examples of occupations General and specialist managers, managing supervisors, business managers Scientists, architects, engineers, medical practitioners, school teachers, social and business professionals Medical and science technical officers, technicians, technical workers, registered nurses, police officers Fitters, mechanics, electrical and electronic tradespersons Graphic artists, photographers, pianists, actors, dancers, desktop publishers Accountants, clerks, data analysts, computer programmers, mathematicians, physicists Authors, journalists, editors, script writers Surgeons, general practitioners, nurses Electrical technicians, mechanical engineers, architects Biologists, chemists, physicists, mathematicians, surveyors, teachers (APTS, 2000, pp. 6 7) Educational and vocational guidance uses The ACER TAR tests may also be used in conjunction with a record of attainment and other data, in counselling a candidate in selecting an appropriate course of study, and in the selection of occupations where further study and training are involved. (See Nelson Bolles, 2002; Holland, 1997; Kummerow, 1991; Anastasi, 1976; Cronbach, 1970.)

10 Test Options Short or long form ACER TAR has two forms: a Short Form (TAR45) and a Long Form (TAR60). Short Form (TAR45) TAR45 is the easier of the two tests and is intended for use with candidates who have completed at least Year 11 and who are applying for a variety of technical, clerical and administrative positions where in-service training, or part-time or after work study is involved. Long Form (TAR60) TAR60 includes the 45 items in the Short Form, plus an additional 15 items. TAR60 is the more challenging test and is appropriate for use with candidates who have completed at least Year 12 and are applying for positions where the work requires a high level of reasoning ability or where these qualities will become increasingly important as staff are promoted in a career system. Table 3 provides examples of appropriate forms for selected jobs. Table 3 Examples of appropriate forms for selected jobs Short Form (TAR45) Personal assistant Market research interviewer Accounts clerk Insurance investigator Customer service operator Call centre operator Workplace trainer Legal clerk Long Form (TAR60) Journalist/copywriter Computer programmer/analyst Market research analyst Financial advisors Graduates Managers Executives Paper or online administration ACER TAR is available for online administration or administration in paper and pencil format. In both cases ACER TAR should always be conducted under appropriate supervision. In no circumstances should ACER TAR be administered in an unsupervised environment. Candidates should not be provided with login codes to access the online test from home or other remote locations unless appropriate supervision has been put in place. Paper Test administrators should provide one ACER TAR reusable test booklet and one ACER TAR answer sheet for each candidate. Completed ACER TAR OMR answer sheets should be sent to ACER Test Scoring Services for scoring and report generation. Alternatively, responses on non-omr answer sheets can be keyed into the Online Response Entry System by the test administrator for immediate access to results. Online Test administrators should provide each candidate with a unique login code, which allows candidates to access the tests online. On completion of the online test, results will be automatically generated and a report ed to the test administrator. Further details regarding online testing are provided in Directions for Administration on p. 13. Contact ACER Press Customer Service on for more information about setting up an online testing account.

11 Scoring and Reporting There are no score keys or norm tables in this manual (apart from those relating to conversion between ACER TAR and Raven s tests on pp. 9 10). ACER TAR can only be scored through one of the automated scoring systems described below. Paper administration ACER TAR can be administered in paper and pencil format using the ACER TAR test booklet and an ACER TAR answer sheet. There are two options for scoring and reporting. ACER Test Scoring Services Information on how to use the ACER Test Scoring Services can be found at ACER TAR OMR answer sheets sent in for test scoring should be accompanied by a copy of the Test Scoring Order Form that comes with orders of the answer sheet. Reports will be provided to test administrators in a format similar to the report shown in Figure 1 below. Candidates can be listed alphabetically or based on test scores. Figure 1 Example of a group report from ACER Test Scoring Services Online Response Entry System Test administrators can manually enter data from non-omr administrations into the online system. ACER TAR Online Response Entry login codes can be purchased from ACER Press Customer Service. Once the response data has been confirmed, the online system will automatically score the responses and generate a report. See the Online administration section (p. 5) for more details on getting access to candidate results.

12 S c o r i n g a n d R e p o r t i n g Online administration Tests completed online are automatically scored by the system. Reports can be obtained in one of three ways: automated reporting; via the Practitioner s site of the ACER Online Testing System; or direct request to ACER Press Customer Service. Automated reporting To obtain automated reports, an address must be supplied at the time of purchasing ACER TAR login codes. The address supplied should be that of the test administrator or HR manager, not the candidate. A separate report (see Figure 2) is generated for each ACER TAR candidate and automatically sent to the test administrator once the responses are submitted online. Figure 2 Example of an automated report ACER Online Testing System Test administrators can set up an Online Testing System account by contacting ACER Press Customer Service on Once this is done, individual or group results can be accessed at any time from the Practitioner s site. An example of how results are displayed is shown in Figure 3 (p. 6). Direct request Test administrators who are conducting large-scale testing may prefer to wait until the last candidate has completed testing and then request ACER Press Customer Service to compile the results of all candidates in the group in a single report. Direct request reports are also available for test administrators using the Online Response Entry System for scoring and reporting of paper tests.

13 A C E R Te s t o f A b s t r a c t R e a s o n i n g Figure 3 Example of a group report from the Practitioner s site.

14 Interpretation of ACER TAR Reports The raw score on a general ability test has little meaning in itself. Usually the first step in interpreting performance is to convert the number of correct answers (raw scores) into terms that will allow for comparison with a reference group who have taken the test (normative data). Normative data for TAR45 have been obtained from a sample of 151 adults in Australia. See Study design, p. 23 for more information about the sample. Normative data for TAR60 have been obtained from a sample of 295 adults at an Australian university. See Study design, p. 32 for more information about the sample. Converted scores in the ACER TAR reports are presented in terms of percentile ranks, stanines and Rasch scale scores. Interpreting norm scores Norm scores such as percentile ranks and stanines (see below) for use with selection instruments such as ACER TAR enable recruitment officers to place candidates in order of merit, and to select an appropriate cut off point below which candidates will be excluded from further consideration. (See Case studies, p. 11 for examples of how this can be done.) Percentile ranks Percentile ranks show the percentage of the reference group with scores below the candidate s score. For example, a percentile rank of 63 means that the candidate has performed better than 63 per cent of the candidates in the reference group; a percentile rank of 23 means that the candidate has performed better than only 23 per cent of the candidates in the reference group. While percentile ranks are useful as a means of locating a candidate s standing relative to the norm group, one drawback is that percentile ranks are not evenly spaced along a measurement scale. Percentile ranks for a large, representative sample tend to be bunched up somewhere around the middle of a bell-shaped distribution, with smaller numbers of candidates towards the upper and lower ends of the distribution. Because small changes in raw scores around the middle of the distribution will have a considerable impact on the percentile rank, it is easy to over-emphasise differences near the middle set of scores and to under-emphasise differences near the extremes where very few candidates are located. Stanine scores To help overcome the problem of using a unit that is not evenly spaced along the measurement scale, stanine scores have been included. In order to derive stanine scores, nine categories are used. Stanines (except for stanines 1 and 9) are of equal length along the variable axis underlying the normal curve. A stanine of 9 indicates a very high level of achievement relative to the reference group while a stanine of 1 indicates a very low relative achievement. Table 4 shows the percentage of the reference group assigned to each stanine score and a descriptor. Table 4 Stanine groups and descriptors Stanine Percentage of group Descriptor 9 4 Very high Above average Average Below average 1 4 Very low

15 A C E R Te s t o f A b s t r a c t R e a s o n i n g Figure 4 Bell-shaped distribution Rasch scores Raw scores (number of correct scores) on the ACER TAR tests are not on an interval scale. In other words, equal numerical differences between raw scores do not represent equal differences in ability. To provide scores on an interval scale, raw scores have been transformed to scale scores using the Rasch measurement model. Measurement errors associated with these scale scores are also reported. The resulting scale is called the TAR Scale. The development of the TAR Scale is outlined in Scale scores, p. 28. Comparison with Raven s Standard Progressive Matrices (SPM) and Advanced Progressive Matrices (APM) The provisional data from the various studies in Technical Information on p. 22 included a comparison of ACER TAR with the Raven s SPM and APM. Rasch modelling of the data collected has allowed the construction of a table for approximate comparisons between TAR45 raw scores and SPM raw scores as well as TAR60 raw scores and APM raw scores. Raw scores on the SPM and APM can be used to generate percentiles and stanines on TAR45 and TAR60 using the norm data available, and vice versa. Table 5 is a simplified version of the full table comparing TAR45 and SPM raw scores that can be found in Appendix C.

16 I n t e r p r e t a t i o n o f A C E R TA R R e p o r t s Table 5 Simple conversion table between ACER TAR45 and Raven s SPM TAR45 raw score TAR45 scale score SPM raw score * Numbers in bold are obtained by linear interpolation

17 A C E R Te s t o f A b s t r a c t R e a s o n i n g Table 6 is a simplified version of the full table comparing TAR60 and APM raw scores that can be found in Appendix F. Table 6 Simple conversion table between ACER TAR60 and Raven s APM TAR60 raw score TAR60 scale score APM raw score

18 Case Studies Example 1: Recruitment for a public sector department A recruiting officer for a public sector department has been put in charge of the graduate selection program. There are 100 initial applicants and the recruiting officer s aim is to reduce that number to approximately 25 for the next stage. Knowing that successful applicants will need to demonstrate a high level of general ability, she decides to screen all applicants with TAR45. The recruiting officer has access to a computer training lab with 25 computer terminals. She arranges for all applicants to attend one of four testing sessions over two days. After the last testing session is completed, she contacts ACER Press Customer Service and requests a group report on the results of the 100 applicants. The spreadsheet is ed to her showing the candidates achievement by percentile rank, stanine and scale score. Using the spreadsheet, the recruiting officer orders the list by percentile rank. In scanning down the list from highest percentile rank to lowest, she notes that selecting a cut off score of the 80th percentile would retain 27 candidates. This number is in line with the number of candidates the recruiting officer was hoping to select for the next stage, and confirms that all the selected candidates demonstrate a high level of general ability. Example 2: Call centre selection A recruitment agency has been asked to fill five vacancies for a special marketing project in a call centre. On advertising the position, 200 suitable applications are received. This number is reduced to 50 by the recruitment agency consultant on reviewing the applicants CVs. The consultant speaks with the manager of the call centre about the job. In their discussion the consultant notes that the level of detail required for this project is relatively high, the training window is relatively short and that information will need to be relayed to customers in a clear, logical and informed manner. The consultant concludes that the successful applicant will need to have a moderately high level of general reasoning ability. The manager indicates that she would like to have the applicants undergo some ability testing and attend a group interview in which various role-play activities will be conducted. From this exercise, 10 applicants will be selected for individual interviews. The consultant invites the 50 applicants to attend the group interview. Before starting the role-play activities, he administers TAR45 to all the applicants in paper and pen format. While the group interviews are being conducted, the consultant directs his assistant to enter the candidates responses into the ACER Online Response Entry System. When all responses are entered, the assistant accesses the ACER Online Testing System and downloads a group report on the results of the 50 applicants. The assistant uses the report to rank the candidates by achievement, and gives the spreadsheet to the consultant. At the completion of the group interview, the consultant makes a list of the 10 most likely candidates for the next stage and a further 10 possibles. He compares this list against the results of the test. He notes that of the list of most likely candidates, two have performed poorly or very poorly on the test, with percentile ranks of 28 or below. The other eight have performed well or very well, with percentile ranks above 70. He looks at the scores of the next list of possibles and sees that two of them have performed very well on the test with percentile ranks above 70. He adds these to the list of most likely candidates for further interviewing. Example 3: Promoting floor staff to store manager A trucking company is looking to promote one of the warehouse floor staff to the position of store manager. In addition to general organisational skills, the store manager must be able to work with basic mathematical concepts in areas such as stock control. Five of the warehouse staff apply for the position, and the general manager decides to interview them all. Three of the five applicants are from a non-english speaking background. A high level of English language proficiency is not required, although an assessment of the candidates general ability would be useful. The HR manager decides 11

19 A C E R Te s t o f A b s t r a c t R e a s o n i n g to administer two tests of general ability: ACER Numerical Select for assessing numerical reasoning skills, and the language-free ACER TAR for assessing general ability. The HR manager asks each of the candidates to come into the office in turn to complete the test session on the computer. Both tests are completed online. As each of the candidates completes the test, the results are automatically generated and ed to the HR manager. When all five applicants have completed the testing, the HR manager transfers their results (percentile ranks and stanines) to a spreadsheet and orders them first by ACER Numerical Select results and then by ACER TAR results. Of the five applicants, two score low or very low, at stanine 3 or below, on both tests. Two of the applicants score well on both tests, at stanines 7 or 8, showing both above average numerical reasoning and abstract reasoning ability. The last candidate scores an average stanine 5 on ACER Numerical Select, but stanine 9 (a very high result the highest of the group) on ACER TAR. The HR manager points out to the general manager that while this last candidate scored significantly lower than the other two on numerical reasoning, the ACER TAR score indicates a very high level of abstract reasoning. This is indicative of the ability to learn new things quickly and engage in strategic thinking. Concluding that specific skills in analysing data can be taught, the general manager and HR manager decide to promote the person with the high ACER TAR score and provide him with training in data analysis using their inventory tracking software. 12

20 Directions for Administration General information ACER TAR Short Form (TAR45) has 45 items and should be completed in 20 minutes. ACER TAR Long Form (TAR60) has 60 items and should be completed in 25 minutes. Please note these details and ensure that the instructions given to candidates are accurate. The data for the interpretation of the test were collected using standardised instructions. It is important that the examiner does not depart from these instructions when administering the test. To achieve this, the following points should be carefully observed: When administering these tests, do not vary in any way from the instructions given in the manual. No candidate should see a test booklet before testing time, and there should be no opportunity for a candidate to see any test booklet other than his/her own, during the testing session. All test booklets should be accounted for at the completion of testing. These are timed tests; it will be necessary to have either a stop watch or an ordinary watch with a second hand. If the latter is used, the starting and finishing time of the testing session should be recorded. The time limits given for the working of each section of the test should be observed exactly. If a test candidate asks a question during the practice items, it should be dealt with as follows: (a) If the question concerns details of procedure, e.g. Where do I write the answers?, it should be answered directly. (b) If the question concerns the explanation of an answer, the relevant instructions should be repeated. Paraphrasing should be used with great caution to ensure that the explanation does not differ in any substantial way from that given in the instructions. (c) Once candidates have started work on the actual test paper, no questions other than procedural questions should be answered. To ensure that the candidates perform at their best, the anxiety level of the candidates should be minimal. Therefore while adhering strictly to the instructions, try to avoid any feeling of strain or strangeness. The physical comfort of the candidates should be given careful consideration; the testing environment should have adequate seating, work space, lighting and ventilation. All practical arrangements should be considered well before the testing session. The following points should be noted: (a) The tester should be familiar both with the directions for administration and with the relevant test booklet well before the testing session. (b) A place for testing should be arranged and the seating and room layout checked. (c) Adequate quantities of test materials should be available. (d) If more than 30 candidates are to be tested, an assistant should be present. (e) Although candidates usually provide their own pens/pencils, the tester is advised to provide a few spares. (f ) Measures should be taken to ensure that external noise is minimal and that the testing session will be completed without interruption. General information for online testing All practical arrangements should be considered well before the testing session. The following points should be noted: A place for testing should be organised with a sufficient number of computers arranged for testing purposes (so that, for example, the test administrator can see all screens, candidates can only see their own screen, etc.) All computers should be checked (screen, keyboard, mouse, browser settings, internet connection, etc.) prior to the testing session. 13

21 A C E R Te s t o f A b s t r a c t R e a s o n i n g If required, an assistant should be present to monitor candidates. This is essential if the number of candidates is above 20. Paper and pencils for working should be provided for each candidate. Measures should be taken to ensure that external noise is minimal and that the testing session will be completed without interruption. During the test sessions, candidates should not use any other program or navigate to any other website other than the testing website. In addition, candidates should only use the buttons in the browser window (e.g. NEXT, PREVIOUS) and not use the web browser s Back or Forward buttons. Consideration should be given to removing software and/or access to programs and websites on the computers that will allow candidates to access information that would assist them in answering the test questions (e.g. calculators, dictionaries, general internet access). In any case, candidates should be closely monitored to ensure that they do not navigate away from the online testing window. Minimum system requirements Users will need access to computers with Internet Explorer version 6 or greater. AutoComplete for forms should be disabled. Sufficient internet connection bandwidth should be available at least 10 kb per second per candidate. This should be checked with the administrator of the internet/network. The requirements for ACER TAR are slightly higher than text-based tests due to the use of image files. In general terms: a single candidate should have the equivalent bandwidth (at a minimum) of a dial-up connection 20 candidates should have the equivalent of a broadband connection (512K). Specific instructions The detailed instructions for paper and online administrations set out exactly what should be done and said. The spoken instructions are enclosed in boxes and should be given verbatim. Do not say anything other that what is indicated in the boxes. 14

22 D i r e c t i o n s f o r A d m i n i s t r a t i o n Instructions for paper administration Make sure all candidates have a pencil, test booklet and answer sheet. Say: The test you will be doing today is a test of observation and problem solving using patterns, shapes and diagrams. On your desk there should be a test booklet and an answer sheet. Hold up the test booklet. Say: This is the test booklet. Please do not open the test booklet until you are told to. Hold up the answer sheet. Say: This is the answer sheet. All your answers are to be marked on the answer sheet. Do not make any marks on the test booklet. Say: The answer sheet you are using will be scored by computer. It is important that you mark it properly. You will see that on the answer sheet there are many oval shapes or bubbles with letters or numbers in them. To give your answers to the test, you need to fill in these bubbles with a pencil. You should fill in the bubble completely, not just mark it with a tick, line, cross or circle. If you change your mind about an answer, you should completely erase the mark you have made and fill in another bubble. To practice using the answer sheet, we will all now fill in our personal details in the top half of the sheet. Hold up the answer sheet. Say: Where the answer sheet says Name, write in your name. Now look across the sheet and print your surname in the squares under SURNAME/FAMILY NAME. Now fill in the bubbles underneath. If your surname is Eade, you would fill in the bubbles E, then A, then D, then E. Now print your initials in the squares under INITS. Now fill in the bubbles underneath. Check that the candidates are correctly filling in the answer sheets. Say: Where the answer sheet says TEST FORM, fill in the bubble for (Short Form/Long Form). Say: Now fill in your sex and date of birth. Please note that this information will be used solely for identification and research purposes. They will not be reported in your results. Wait for a few seconds. Hold up the test booklet. Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

23 A C E R Te s t o f A b s t r a c t R e a s o n i n g Say: We are now ready to look at the test booklet and do some practice questions. Please open your test booklet to the first page. Do not turn the page until I tell you to. Hold up the test booklet open to the first page. Say: Look at the first practice question P1. The instruction says: The three shapes in the top row are alike in some way. Which shape in the bottom row is most like them? You will see that the five shapes in the bottom row each have a letter above them. On your answer sheet, look for the shaded rectangle that says Practice Examples. Next to P1, fill in the bubble with the letter of the shape you think is most like the top three shapes. Wait for a few seconds. Say: The correct answer is shape A, so you should have filled in the bubble A next to P1 on the answer sheet. Remember not to mark the test booklet, only the answer sheet. Now have a look at the second practice question P2. The instruction says: On the top row there are five squares with shapes arranged in order. One shape is missing. Which shape from the bottom row best completes the pattern? On the answer sheet next to P2 fill in the bubble with the letter of the shape you think best completes the pattern. Wait for a few seconds. Say: The correct answer is shape E, so you should have filled in the bubble E next to P2 on the answer sheet. Now have a look at the third practice question P3. The instruction says: There is one missing shape in the pattern at the top. Which shape from the bottom row best completes the pattern? On the answer sheet next to P3 fill in the bubble with the letter of the shape you think best completes the pattern. Wait for a few seconds. Say: The correct answer is shape A, so you should have filled in the bubble A next to P3 on the answer sheet. That completes the practice questions. A few things to remember: Fill in your answer on the answer sheet, not the test booklet When you choose an answer, fill in the whole bubble on the answer sheet don t tick, cross or circle the answer 16 Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

24 D i r e c t i o n s f o r A d m i n i s t r a t i o n If you change your mind about an answer, erase the mark you have made and fill in the bubble of your new choice. Are there any questions? Answer any questions about filling in the answer sheet. If you are administering TAR45, say: In the test booklet there are 60 questions. Today we are only doing the first 45 questions. You should finish at question 45 where it says END OF SECTION 1. Hold up the answer sheet and point to the bottom half of the sheet where the candidates fill in their answers. Say: You can see on the answer sheet that there is space for answers to 60 questions. We will only fill in the first 45 down to where it says END OF SECTION 1. Are there any questions? If you are administering TAR60, say: In the test booklet there are 60 questions. After 45 questions, there is a message END OF SECTION 1. Please ignore that message and work through the test booklet all the way to the last page. Answer any questions about the number of items to be completed. Hold up the test booklet. Say: Pause. In Section 1 of the test booklet there are five sets of questions. The instruction for the questions is given once at the beginning of the set. So for each instruction there are several questions. Each new set of questions has a new instruction for all the questions in that set. If you are administering TAR45, say: You will have 20 minutes to complete all the questions. Please note that there should be no talking during the test. If you need anything or want to ask a question, just raise your hand and one of the supervisors will help you. Don t spend too much time on any one question try to answer as many questions as you can in the time. After 20 minutes I will tell you to stop writing. Please put down your pencils and wait for me to collect your answer sheet. Before we begin, are there any questions? If you are administering TAR60, say: You will have 25 minutes to complete all the questions. Please note that there should be no talking during the test. If you need anything or want to ask a question, just raise your hand and one of the supervisors will help you. Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

25 A C E R Te s t o f A b s t r a c t R e a s o n i n g Don t spend too much time on any one question try to answer as many questions as you can in the time. After 25 minutes, I will tell you to stop writing. Please put down your pencils and wait for me to collect your answer sheet. Before we begin, are there any questions? Answer any questions. Say: We are now ready to begin the test. You have (20/25) minutes to complete the test, please turn the page now and begin. If administering TAR45, wait 20 minutes. If administering TAR60, wait 25 minutes. When the time limit is up, say: Please stop writing and put your pencils down. Collect the test booklets and answer sheets. Make sure that each candidate has correctly filled in their details on the answer sheet. 18 Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

26 D i r e c t i o n s f o r A d m i n i s t r a t i o n Instructions for online administration Make sure all candidates have a login code and a pencil and some paper for working. All the candidates computers should already display the ACER Press Online Testing System login page. Say: Please enter your login code and click the ENTER button. or you may enter the login codes into the screens and click the ENTER button prior to the candidates arrival. Say: Please fill in the personal details requested. Do not click submit until you are told to do so. Please note that the personal details requested are used for identification and research purposes only and will not be provided along with test results. Allow time for filling in the personal details. When all candidates have filled them in, say: Now click the SUBMIT button (Pause). The test you will be doing today is a test of observation and problem solving using patterns, shapes and diagrams. If you are administering TAR45, say: Look at the screen and follow while I read. This test is a timed test and will take 20 minutes. There are 45 questions in this test. There are some practice questions to complete before the test begins. If you are administering TAR60, say: Look at the screen and follow while I read. This test is a timed test and will take 25 minutes. There are 60 questions in this test. There are some practice questions to complete before the test begins. Say: Now click on BEGIN PRACTICE QUESTIONS. On the next few screens you will see a number of practice questions to help you in the test that follows. All the questions require choosing one answer from the list of possible answers. To answer each question, click the small circle below the answer. Look at the first practice question. The instruction says: The three shapes in the top row are alike in some way. Which shape in the bottom row is most like them? You will see that the five shapes in the bottom row each have a small circle below them. Click on the circle under the shape you think is most like the top three shapes. The circle should now have a dark spot inside it. Wait for a few seconds. Say: The correct answer is the first shape, so you should have clicked the small circle under the first shape. Now click NEXT and have a look at the second practice question. The instruction says: Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

27 A C E R Te s t o f A b s t r a c t R e a s o n i n g On the top row there are five squares with shapes arranged in order. One shape is missing. Which shape from the bottom row best completes the pattern? Click the small circle under the shape you think best completes the pattern. Wait for a few seconds. Say: The correct answer is the fifth shape, so you should have clicked the small circle under the fifth shape. Now click NEXT and have a look at the third practice question. The instruction says: There is one missing shape in the pattern at the top. Which shape from the bottom row best completes the pattern? Click the small circle under the shape you think best completes the pattern. Wait for a few seconds. Say: The correct answer is the first shape, so you should have clicked the small circle under the first shape. That completes the practice questions. Click NEXT and wait. Do not click ahead until I tell you to. Say: In the test there is an instruction for each of the questions. Several questions in a row will have the same instruction, then there will be a change of instruction. Be careful to note when the instruction changes. If you are administering TAR45, say: You will have 20 minutes to complete all the questions. If you are administering TAR60, say: You will have 25 minutes to complete all the questions. Say: Please note that there should be no talking during the test. If you need anything or want to ask a question, just raise your hand and one of the supervisors will help you. Don t spend too much time on any one question try to answer as many questions as you can in the time. Before we begin, are there any questions? Answer any questions. Say: We are now ready to begin the test. 20 Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

28 D i r e c t i o n s f o r A d m i n i s t r a t i o n During the test, use only the NEXT and PREVIOUS buttons under each question or the numbered navigation panel on the left hand side of the screen to move through the test. Under no circumstances use the Back or Forward buttons on your browser or the backspace button on your keyboard. If you finish ahead of time, you may go back over your answers if you wish or click the FINISH button. During the test you may not use any other software or programs on the computer or navigate away from the testing system website. If you are administering TAR45, say: You will have 20 minutes to complete the test. Please click on BEGIN TEST. If you are administering TAR60, say: You will have 25 minutes to complete the test. Please click on BEGIN TEST. If administering TAR45, wait approximately 20 minutes. If administering TAR60, wait approximately 25 minutes. The system is timing each individual but you should also keep an independent watch on the time. NOTE: If a candidate goes over time but does not click NEXT, PREVIOUS or FINISH on the item they are currently working on, it may appear that the system has not timed them out. When they eventually click NEXT, PREVIOUS or FINISH, the system will time the session out and close the test. In such a case when you are sure the time has gone past the time limit, ask the candidate to click one of the buttons. When all the candidates have been timed out, say: The time limit is up. Please click on the FINISH button to submit your responses. Co p y r i g h t A u s t r a l i a n Co u n c i l for E d u c a t i o n a l R e s e a r c h

29 Technical Information Users of ACER TAR should note that the study samples reported in this manual are not stratified samples of the Australian population. However, percentiles, stanines and scale scores have been derived from these reference groups to provide a point of comparison with an individual candidate s performance on the test. Demographic details of the samples used in each study are provided. Summary of research studies Study 1 was conducted on 151 adult volunteers from university, TAFE campuses, Centrelink and Job Network Providers using mainly TAR45 paper and pencil tests. Key descriptive, reliability and validity findings are given in the table below. Table 7 Key statistical findings in Study 1 Statistic Mean raw score Value out of 45 items Std. deviation 6.58 Internal reliability Correlation with SPM Demographic variable(s) that showed a significant difference in test scores 0.87 (Good) 0.80 (High) Age groups Study 2 was conducted on 295 adults using mainly TAR60 paper and pencil tests. Key descriptive, reliability and validity findings are given in the table below. Table 8 Key statistical findings in Study 2 Statistic Mean raw score Value out of 60 items Std. deviation 6.71 Internal reliability Correlation with APM Demographic variable(s) that showed a significant difference in test scores 0.84 (Good) 0.76 (High) Highest level of education completed Data for Study 3 was obtained from a total of 329 adult jobseekers who had completed TAR45 online. Key descriptive and reliability findings are given in the table below. Table 9 Key statistical findings in Study 3 Statistic Mean raw score Value 36.6 out of 45 items Std. deviation 7.29 Internal reliability