Participant Tracking in Text Unfolding: Insights from Portuguese-Chinese Translation and Post-Editing Task Logs

Transcription

1 Participant Tracking in Text Unfolding: Insights from Portuguese-Chinese Translation and Post-Editing Task Logs AuTema-PostEd Group Ana Luísa V. Leal 1, Márcia Schmaltz 1, Derek Wong 1, Lidia Chao 1, James TL Wang 1, Adriana Pagano 2, Fabio Alves 2, Igor da Silva 3, Paulo Quaresma 4 University of Macau (UM) 1 Federal University of Minas Gerais (UFMG) 2 Federal University of Uberlândia (UFU) 3 University of Evora (UE) 4

2 Outline Research Context and Aims Research Assumptions and Questions Experimental Design Methodology of Analysis Preliminary Results Discussion Next Steps

3 Research Context and Aims Joint Project between Department of Portuguese /University of Macau and LETRA Laboratory for Experimentation in Translation, Federal University of Minas Gerais, to investigate the process and final output of human translated text as compared to human postediting of machine translated texts by Portuguese-Chinese Translation System (PCT).

4 Research Assumptions and Questions Identity cohesive chains responsible for co-reference and participant tracking are crucial to construing a coherent representation of a text. (Halliday & Hasan 1976, Halliday 1989, Halliday & Matthiessen 2004) Do translation process data show evidence of the role of identity chains as exerting cognitive demands upon task executors? Translating demands more cognitive effort than post-editing (Carl et al 2011) Do data in our study confirm impact of task type on effort?

5 Translation Process Research Eye-Mind Assumption (Just & Carpenter 1980) User Activity Data (Carl & Jakobsen 2009, Carl 2012b) (Un)challenge translation (Carl & Dragsted 2012) More / less time consuming Look few / more words ahead into the ST More / less keyboard activities (insertions, deletions) Long pauses, regressive saccades and refixation on words already read (Carl & Jakobsen 2009) Integrating quantitative and qualitative analysis to have a full picture of the translation process (Alves & Gonçalves 2013, O Brien 2006)

6 Experimental Design Materials Translog-II (Carl 2012a) Eye tracker Tobii T120 Tobii Studio Participants 12 translators, L1 Chinese, years old, BA Portuguese Studies, < 1 year experience, glasses or contact lenses Setting and conditions Eye-tracking lab at University of Macau No time pressure

7 Experimental Design Input 4 texts: ca. 80 words/word-equivalents news reports 2 Chinese source texts 2 Portuguese source texts MT output provided by PCT (Portuguese-Chinese Translator) (Wong et al 2012) Randomized for both subjects and tasks Tasks 1 L1 translation 1 L2 translation 1 post-editing into L1 1 post-editing into L2 Recall Protocols

8 Experimental Design Input 4 texts: ca. 80 words/word-equivalents news reports 2 Chinese source texts 2 Portuguese source texts [Focus on Text 2] MT output provided by PCT (Portuguese-Chinese Translator) (Wong et al 2012) Randomized for both subjects and tasks Tasks 1 L1 translation 1 L2 translation 1 post-editing into L1 1 post-editing into L2 Recall Protocols

9 Os brasileiros estão em lua-de-mel com o mundo. The Brazilians are in honeymoon with the world. A Petrobrase sua presidentee sua presidente estão entre as maiores do mundo. Petrobras and their/its president are among the greatest of the world. Conforme uma revista americana, a presidenteé uma das 100 maiores lideranças mundiais e a petrolífera estatal é uma das 10 maiores companhias do planeta. According to an American magazine, the president is one of the 100 greatest world leaders, and the stateowned oil companyis one of the 10 greatest companies in the planet. A revista classifica as empresas com base em diversos indicadores além do lucro. The magazine classifies the companies building on several indicators besides profit. É por isso que a brasileira surge em posição de liderança na classificação. That s why the Brazilian[company/president] emerges in leadership position in the classification. Input ST Identity Cohesive Chains [Elipse] Está inclusive à frente de grandes como a Apple. [The company/brazil/the president]are even beating big [ones/companies/countries/presidents], such as Apple.

10 Os brasileiros estão em lua-de-mel com o mundo. The Brazilians are in honeymoon with the world. A Petrobrase sua presidentee sua presidente estão entre as maiores do mundo. Petrobras and their/its president are among the greatest of the world. Conforme uma revista americana, a presidenteé uma das 100 maiores liderançasmundiaise a petrolífera estatal é uma das 10 maiores companhias do planeta. According to an American magazine, the president is one of the 100 greatest world leaders, and the stateowned oil companyis one of the 10 greatest companies in the planet. A revista classifica as empresas com base em diversos indicadores além do lucro. The magazine classifies the companies building on several indicators besides profit. É por isso que a brasileira surge em posição de liderança na classificação. That s why the Brazilian[company/president] emerges in leadership position in the classification. Input ST Identity Cohesive Chains [Elipse] Está inclusive à frente de grandes como a Apple. [The company/brazil/the president]are even beating big [ones/companies/countries/presidents], such as Apple.

11 Methodology of Analysis User Activity Data FU 400ms, PU 1000ms Source Tokens (ST) Target Tokens (TT) Extracted: gaze time, fixation number, insertions, deletions, editions, first pass fixation time (FD), STid, TTid. Statistical studies Comprehension of ST, TT and Production Subset studies with the 6 identities references (Principal, Secondary, Others)

12 Methodology of Analysis Quantitative Linear Mixed-Effects Regression Model (LMER) lmertest running on the statistical tool R (3.0.2) (Baayen 2008, Balling 2008, 2013, Sjorup 2012) p<0.05 Tests (Student s T-Test; Mann Whitney U)

13 Methodology of Analysis Variables Comprehension Production Dependent Total Fixation Time Total Fixation Number First Pass Fixation Time Production time Independent / Random Independent / Fixed AOIs, Participant Length Position Frequency (corpus Banco de Português) Trigram Probability Task (Translation or Post-editing) Type (Reference or Comparative) AOIs, Participant Character count Position Rendition (Yes or No) Frequency (corpus CCL/UPK) Trigram Probability Task (Trans. or Post-Ed.) Type (Reference or Comparative)

14 Preliminary Results

15 Source Text Comprehension Gazing and Fixation Variables Total Fixation Time Total Fixation Number First Pass Fixation Time (Intercept) <2e <2e-16 Length <2e-16 <2e Frequency

16 Target Text Comprehension Gazing and Fixation Variables Total Fixation Time Total Fixation Number First Pass Fixation Time (Intercept) <2e e-08 <2e-16 Length 6.90e e e-10 Position 5.62e e-09 TypeRef

17 Target Text Production Keylogging Variables Production Time (Intercep) <2e-16 Character count 2.67e-13 Renditions

18 Os brasileiros estão em lua-de-mel com o mundo. The Brazilians are in honeymoon with the world. A Petrobrase sua presidentee sua presidente estão entre as maiores do mundo. Petrobras and their/its president are among the greatest of the world. Conforme uma revistaamericana, a presidenteé uma das 100 maiores lideranças mundiais e a petrolífera estatal é uma das 10 maiores companhias do planeta. According to an American magazine, the president is one of the 100 greatest world leaders, and the stateowned oil companyis one of the 10 greatest companies in the planet. A revista classifica as empresas com base em diversos indicadores além do lucro. The magazine classifies the companies building on several indicators besides profit. É por isso que a brasileira surge em posição de liderança na classificação. That s why the Brazilian[company/president] emerges in leadership position in the classification. Input ST Identity Cohesive Chains [Elipse] Está inclusive à frente de grandes como a Apple. [The company/brazil/the president]are even beating big [ones/companies/countries/presidents], such as Apple.

19 ST Identity Chains Gazing and Fixation Variables Total Fixation Time Total Fixation Number First Pass Fixation Time (Intercept) <2e e-09 <2e-16 Length e Frequency

20 TT Identity Chains Gazing and Fixation Variables Total Fixation Time Total Fixation Number First Pass Fixation Time (Intercept) <2e e Length 3.16e e Position 8.20e e-10 Frequency 7.35e-05

21 Production Identity Chains Keylogging Variables Production Time (Intercep) <2e-16 Character count 1.72e-08 Renditions Position

22 Non-Parametric Tests ST Reading: Significant differences between reference types for TNF (0.27) and TFT (0.46) and between subjects for TNF, TFT and FIRST No significant differences between tasks TT Reading: Significant differences between subjects for TNF, TFT and FIRST Significant differences between tasks for TNF (0.13) and TFT (0.025) No significant differences between reference types

23 ProgGraph P23 Some translators take the wrong road when accepted the MT or due the comprehension of the ST 以 及 巴 西 裁 统

24 Rereading and... click! Prograph P 统 总 西 巴 其 总 裁

25 Discussion Research question: Do translation process data show evidence of the role of identity chains as exerting cognitive demands upon task executors? They do with regard to the source text considering only the non-parametric tests. The multiple regression tests show no significant results considering the set of variables used in the experiment. Cohesive chains seem to have an impact on comprehension when reading the ST reading for translation involves anticipating how chains will have to be dealt with in the TT (whether explicitation will be needed in the TT)

26 Discussion Research question: Do data in our study confirm impact of task type on effort? No. However four subjects built identity chains different from those in the ST. Their inferential path was different from the other subjects The results of non-parametric tests shed light on relevant aspects of the inferential processing in post-editing and translation Translation TT-driven, but a result of a comprehension process Tasks differ concerning the target text production Post-editing demands less cognitive effort for lexical rendition (as confirmed by RVP, i.e. less character insertions), but requires more effort to reorganize structures (as shown in RVP)

27 Next Steps Fine-grained, qualitative analysis of user activity data (UAD) and translation progression graphs Analysis of the impact of the first task on the second task (Ferreira 2010) Further studies collecting data from native speakers of Portuguese to contrast reading effort in cohesive chains. Analyses of results for all four texts used in the experiment will permit more robust results.

28 References ALVES, F., GONÇALVES, J Investigating the Conceptual-procedural Distinction in the Translation Process. Target 25:1. p BALLING, L., CARL M. (to appear). Production Time Across Language and Tasks: A Large-scale Analysis Using the CRITT Translation Process Database. In: Schwieter, J., Ferreira, A. (eds.) The development of Translation Competence: Theories and Methodologies from Psycholinguistics and Cognitive Science. Cambridge: Cambridge Scholar Publishing. CARL, M Translog-II: a program for Recording User Activity Data for Empirical Reading and Writing Research, in Proceedings of the Eighth International Conference on Language Resource and Evaluation. Istanbul May 2012, Istanbul: European Language Resources Association, CARL M., DRAGSTED, B., ELMING, J., HARDT, D. & JAKOBSEN, A. L The Process of Post-Editing: a Pilot Study. In B. Sharp, M. Zock, M. Carl, A.L. Jakobsen (orgs.). Proceedings of the 8th Natural Language Processing and Cognitive Science Workshop. Copenhagen Studies in Language Series 41. p HALLIDAY, M. A.K., HASAN, R Cohesion in English. London: Longman. HALLIDAY, M. A.K Spoken and Written Language. Oxford: Oxford University Press. HALLIDAY, M. A.K., Matthiessen, C An Introduction to Functional Grammar. London: Arnold O'BRIEN, S Teaching post-editing: A proposal for course content. In: Teaching Machine Translation - the 6th International Workshop of the European Association of Machine Translation, Centre for Computational Linguistics, UMIST: Manchester. p O`BRIEN, S Machine translatability and post-editing effort: how do they relate. In: Proceedings of translating and the computer 26. London: Aslib. O BRIEN, S Controlled Language and Post-Editing. The Guide From Multilingual. p O BRIEN, S. & ALMEIDA, G Analysing post-editing performance: correlations with years of translation equivalence. In: Proceedings of EAMT 2010: the European Association for Machine Translation, St Raphael, France. PAVLOVIĆ, N. & JENSEN, K. T. H Eye tracking translation directionality. In A. Pym and A. Perekrestenko (eds). Translation Research Projects 2. Tarragona: Universitat Rovira i Virgili. p SJORUP, A. (2013). Cognitive Effort in Metaphor Translation. PhD Thesis. Copenhagen: Copenhagen Business School. WONG, D., OLIVEIRA, F., LI, YP Hybrid Machine Aided Translation System based on Constraint Synchronous Grammar and Translation Corresponding Tree. Journal of Computers, 7(2): p

29 Thank you! Obrigada! 谢 谢!