Challenges of Automation in Translation Quality Management

Transcription

1 Challenges of Automation in Translation Quality Management Berlin, Dr. François Massion D.O.G. Dokumentation ohne Grenzen GmbH

2 Overview Quality definition and quality standards Requirements and current practice Origins of quality problems Automation potential and challenges Outlook Wir nehmen Qualität beim Wort. 5. Oktober

3 What is quality? Actual translation Technical aspects (formats, hyperlink, software) Adherence to deadlines Service Organization (archiving, maintenance of translation memories, terminology, delivery ) Different assessments of what is good and bad Wir nehmen Qualität beim Wort. 5. Oktober

4 Some translation related standards SAE J 2450 Translation Metrics SAE = Society of Automotive Engineers (USA), under development since 1997 Working group: GM, Ford, Chrysler + LSP Target: measurable quality Error categories + Weighting factor per category Sum of points / Word numbers = Quality index Focused primarily on technical texts such as service manuals for vehicles LISA QA Model 3.0 EN European Quality Standard (four-eyes principle) Tekom quality criteria (working group, 2005) Wir nehmen Qualität beim Wort. 5. Oktober

5 Challenges in global organizations Overall challenges: Global competition shorter product development cycles = need to react faster ( time to market ) & more frequent updates More product variants information maintenance & costs Growing number of foreign languages more resources, higher administrative costs Increasingly complex information technologies CMS, XML, data bases, workflow system Effects on documentation and translations: Chunks of documents = reduced context information, inconsistencies CAT systems = little context MT technologies = linguistic quality, style Complex terminology not always managed Multiple suppliers = varying quality, maintenance of translation memories and terminology Lack of universally accepted quality standards Wir nehmen Qualität beim Wort. 5. Oktober

6 Origin of quality problems Linguistical Quality of the source documentation Semantics e.g. ambiguity Syntax Technical Mode of operation of translation memories Segmentation Fuzzy Matches Organizational Workflow (order of tasks, schedule, chunks of information translated) Selection of translators Communication Wir nehmen Qualität beim Wort. 5. Oktober

7 Translation memories and quality Context is missing Undiscovered errors propagate themselves (as 100% matches) Segmentation issue (incomplete sentences) Fuzzy matches can lead to false translations (confirmed too fast) 99% Wir nehmen Qualität beim Wort. 5. Oktober

8 Today s practice Quality assurance is mostly a manual process Quality assurance not a high priority in many organizations: The translator is in charge, I haven t heard anything negative... Spot checks, uncritical reuse of existing translations (TM) Often restricted to formal aspects such as spelling or layout Large terminologies (5,000+ terms) not really checked Less familiar languages are rarely checked. Key problem: time & costs of a thorough and systematic quality assurance Wir nehmen Qualität beim Wort. 5. Oktober

9 Allocation of translation costs 20% 10% 10% 10% Project Management & Communication Terminology Translation Quality Assurance 50% Technical Preparation and Memory maintenance QA costs = approx. 20% of total translation costs (if compliant with EN requirements) 50.- per hour and Ø 8,000 words per day = 0.05 per word (approx. 20% variance) Wir nehmen Qualität beim Wort. 5. Oktober

10 Limitations of human QA Problems with large volumes: Time: reviewer 6,000 10,000 words per day Resources: 1,000 pages per day in 30 languages? Problems with the results: Terminology: how to verify 5,000+ terms?! Accuracy of formal checks: numbers, typography, completeness Consistency: work by teams of translators Check of unknown/less known languages Problems with feedback and evaluation: Immediate feedback to translators? Objective + reproducible evaluation of quality? Wir nehmen Qualität beim Wort. 5. Oktober

11 Limitations of automatic QA It only detects formal error patterns (compares strings with preset values) No intelligent checks requiring judgment (e.g. misunderstanding of source text) Level of noise (false error messages) Solution: Automatic QA for formal checks, Human QA for checks requiring intelligence. Wir nehmen Qualität beim Wort. 5. Oktober

12 Which checks can be automated? Quality issue Automatic Manual ErrorSpy since 2003 Completeness Generates automatically list of errors Consistency Numbers Computes automatically a quality index (evaluation of translation) Style Tags Terminology Typography Understanding Wir nehmen Qualität beim Wort. 5. Oktober

13 How can quality be measured? Error categories + Weighting factor per category Sum of error points / Number of words = Quality index ErrorSpy reports use metrics adapted from SAE J2450 Can be used for vendor assessment: Objective evaluation Reproducible assessment of vendors (configuration of checks saved as a template) Wir nehmen Qualität beim Wort. 5. Oktober

14 Example: translation workflow at Siemens Automated workflow since end of 2004 Situation was/is: High volumes Many languages Many suppliers (MLV, SLV) Goal and requirements were: Check linguistically 100% of translation projects Standardize Automate Automatic quality checks (QCs) with instant reports Predefined error levels Actions triggered by report results (Quality gates) One measurable, reproducible, objective quality Pilot project completed in Dec and continuously improved since Wir nehmen Qualität beim Wort. 5. Oktober

15 Implementation of automated quality assurance Quality gates 4 Quality gates: Automated quality evaluation triggers actions in translation workflow system (SDL Teamworks) A = Pretranslation SL1 = Translation SL2 = Proofreading SL3 = Verification (in-country review) (Check results from in-country review) Slide from common presentation at the Tekom conference in Wiesbaden, Germany, Autumn 2007; Francois Massion (D.O.G.), Thomas Hecht (Siemens) Wir nehmen Qualität beim Wort. 5. Oktober

16 Automatic corrections Some types of errors are formal and not language dependant Example: Some consistency errors Some number errors Special case: terminology correction of UI terms. Term is invariable and tagged as e.g.: <menuitem>exit</menuitem> Better to correct 10% of errors automatically in a reliable way than to correct 0% Possibility to log and undo the automatic corrections Wir nehmen Qualität beim Wort. 5. Oktober

17 Quality monitoring Ø errors per period The issue is not just the quality of one project Need to develop metrics to measure the efficiency of QA policy at the corporate level. Type of errors Origin of errors Errors per language combination Errors per supplier Average number of errors per period Costs of repairs etc. Allocation of errors according to type and period Single project reports (Database) Wir nehmen Qualität beim Wort. 5. Oktober

18 Outlook More automated and instant QA of large volumes of translations Solution for both human and machine translation Improved learning capability (reduction of noise level) Automatic correction of some errors Prevention of errors (feedback to authors, improvement of source documentation) Worldwide accepted standards & metrics for translation quality Web 2.0 cooperation models (e.g. interaction between author, translator, terminologist and reviewer) Wir nehmen Qualität beim Wort. 5. Oktober

19 Thank you for your attention! Contact: François MASSION Internet: Tel.: +49 (0) 7152 / Wir nehmen Qualität beim Wort. 5. Oktober