Dan Joyce Eiken Foundation of Japan

Size: px

Start display at page:

Download "Dan Joyce Eiken Foundation of Japan"

Rodger Chambers
10 years ago
Views:

1 Dan Joyce Eiken Foundation of Japan

2 2

3 1 What is TEAP? 2 TEAP Speaking Test 3 Test Administration Issues 4 Computer Tablet Application 5 Usability Studies 6 Summary 3

4 a new test of academic English for university entrance purposes in Japan Four skills EFL context of Japan Japanese high school students CEFR A2 B2 Model for revision 4

5 Eiken Sophia University CRELLA Centre for Research in English Language Learning and Assessment : Dr. Fumiyo Nakatsuhara 5

6 Part Task Level Language functions 1 Interview A2 -Providing specific personal information 2 Role play A2/B1 3 Monologue B1/B2 4 Extended interview B2 -Initiating interaction -Asking for information/opinions -Commenting -Agreeing/disagreeing -Justifying opinions -Elaborating -Expressing opinions -Justifying opinions -Comparing -Speculating -Elaborating 6

interaction -Asking for information/opinions -Commenting -Agreeing/disagreeing

7 Task Preparation time Response time Part 1-2 min. Part 2 30 sec. 2 min. Part 3 30 sec. 1 min. Part 4-4 min. 7

8 Phasing out of the National Center Test Ministry of Education policy to make use of external English tests for university entrance purposes in Japan Increase in Test Takers Emphasis on need for tests that measure four skills 8

for university entrance purposes in Japan Increase in

9 Maintaining accessibility in terms of cost Ensuring timely score reporting Maintaining reliable scoring a) Double rating methods and rating flow b) Simplifying the examiners interlocuting procedures.

10 Test Day: Examiner assigns holistic score Post-test rating session: Rater 1 assigns analytic scores Examiner score and Rater 1 overall score compared Significant discrepancy Little or no discrepancy Rater 2 (veteran/senior rater) assigns analytic scores Use Rater 1 analytic scores 10

score compared Significant discrepancy Little or no discrepancy Rater

11 Check test taker name/id Control timing for each section of the test (2 timers, 5 functions!) Carry out test following strict test procedures (including following the script in the examiner frame) Assign a score to each test taker Take a back up audio recording of each test (using IC recorder) 11

) Carry out test following strict test procedures (including following the

12 TEAP Speaking Test Examiners needed: Eight arms preferred An ability to multi-task essential Heavy load on examiners Possible negative impact on test takers 12

13 Intensive training required not sustainable Risk of system failure Risk of human error 13

14 Reads QR code on admission ticket Automatically inputs and records test taker ID Provides on-screen interface for each of the four parts of the test Times each part of the test (preparation time and response time) Provides simple procedural instructions for examiners Provides an interface for examiners to input scores Makes an audio recording of each test

(preparation time and response time) Provides simple procedural instructions for

15 15

16 16

17 17

18 18

19 19

20 20

21 21

22 22

23 23

24 24

25 Data collection Examiner feedback questionnaire (following test administration) 13 trained examiners Question examples: Q32 Were the ipad instructions easy to understand? Q43 Were you able to use the ipad to read the QR codes without any problems? Q48 Did you find that using the ipad interfered with your ability to deliver the tests? 25

26 Examiners generally found the app easy to use (12/13) Examiners said using the ipad did not interfere with their ability to deliver the tests (13/13) However Some (3/13) examiners said they experienced difficulty scanning the QR code it was extremely difficult to scan the square code on the paper. Modifications to app 26

27 Modifications to app Digital finder image added to the QR code-scanning interface Instructions added to examiner training: Input test taker ID manually if necessary 27

28 28

29 The use of a bespoke ipad application has simplified the work of Examiners and facilitates test administration procedures. The use of an ipad has been generally well received by examiners. But on-going monitoring is required! 29

30 For a full validation report: Nakatsuhara et al (2014) peaking_report2.pdf For more information about TEAP: Thank You! [email protected] 30

31 31

32 32

The Japan Society of Mechanical Engineers C 2010

The Japan Society of Mechanical Engineers C 2010 The Japan Society of Mechanical Engineers 2010 C The Japan Society of Mechanical Engineers 2010 C The Japan Society of Mechanical Engineers C 2010 The Japan