Dan Joyce Eiken Foundation of Japan
2
1 What is TEAP? 2 TEAP Speaking Test 3 Test Administration Issues 4 Computer Tablet Application 5 Usability Studies 6 Summary 3
a new test of academic English for university entrance purposes in Japan Four skills EFL context of Japan Japanese high school students CEFR A2 B2 Model for revision 4
Eiken Sophia University CRELLA Centre for Research in English Language Learning and Assessment : Dr. Fumiyo Nakatsuhara 5
Part Task Level Language functions 1 Interview A2 -Providing specific personal information 2 Role play A2/B1 3 Monologue B1/B2 4 Extended interview B2 -Initiating interaction -Asking for information/opinions -Commenting -Agreeing/disagreeing -Justifying opinions -Elaborating -Expressing opinions -Justifying opinions -Comparing -Speculating -Elaborating 6
Task Preparation time Response time Part 1-2 min. Part 2 30 sec. 2 min. Part 3 30 sec. 1 min. Part 4-4 min. 7
Phasing out of the National Center Test Ministry of Education policy to make use of external English tests for university entrance purposes in Japan Increase in Test Takers Emphasis on need for tests that measure four skills 8
Maintaining accessibility in terms of cost Ensuring timely score reporting Maintaining reliable scoring a) Double rating methods and rating flow b) Simplifying the examiners interlocuting procedures.
Test Day: Examiner assigns holistic score Post-test rating session: Rater 1 assigns analytic scores Examiner score and Rater 1 overall score compared Significant discrepancy Little or no discrepancy Rater 2 (veteran/senior rater) assigns analytic scores Use Rater 1 analytic scores 10
Check test taker name/id Control timing for each section of the test (2 timers, 5 functions!) Carry out test following strict test procedures (including following the script in the examiner frame) Assign a score to each test taker Take a back up audio recording of each test (using IC recorder) 11
TEAP Speaking Test Examiners needed: Eight arms preferred An ability to multi-task essential Heavy load on examiners Possible negative impact on test takers 12
Intensive training required not sustainable Risk of system failure Risk of human error 13
Reads QR code on admission ticket Automatically inputs and records test taker ID Provides on-screen interface for each of the four parts of the test Times each part of the test (preparation time and response time) Provides simple procedural instructions for examiners Provides an interface for examiners to input scores Makes an audio recording of each test
15
16
17
18
19
20
21
22
23
24
Data collection Examiner feedback questionnaire (following test administration) 13 trained examiners Question examples: Q32 Were the ipad instructions easy to understand? Q43 Were you able to use the ipad to read the QR codes without any problems? Q48 Did you find that using the ipad interfered with your ability to deliver the tests? 25
Examiners generally found the app easy to use (12/13) Examiners said using the ipad did not interfere with their ability to deliver the tests (13/13) However Some (3/13) examiners said they experienced difficulty scanning the QR code it was extremely difficult to scan the square code on the paper. Modifications to app 26
Modifications to app Digital finder image added to the QR code-scanning interface Instructions added to examiner training: Input test taker ID manually if necessary 27
28
The use of a bespoke ipad application has simplified the work of Examiners and facilitates test administration procedures. The use of an ipad has been generally well received by examiners. But on-going monitoring is required! 29
For a full validation report: Nakatsuhara et al (2014) http://www.eiken.or.jp/teap/group/pdf/teap_s peaking_report2.pdf For more information about TEAP: https://www.eiken.or.jp/teap/ Thank You! d-joyce@eiken.or.jp 30
https://en.alohaworks.jp/blog/2015/01/06/teap-speaking/ 31
32