Challenges of Automation in Translation Quality Management Berlin, 12.09.2009 Dr. François Massion D.O.G. Dokumentation ohne Grenzen GmbH francois.massion@dog-gmbh.de
Overview Quality definition and quality standards Requirements and current practice Origins of quality problems Automation potential and challenges Outlook Wir nehmen Qualität beim Wort. 5. Oktober 2009 2
What is quality? Actual translation Technical aspects (formats, hyperlink, software) Adherence to deadlines Service Organization (archiving, maintenance of translation memories, terminology, delivery ) Different assessments of what is good and bad Wir nehmen Qualität beim Wort. 5. Oktober 2009 3
Some translation related standards SAE J 2450 Translation Metrics SAE = Society of Automotive Engineers (USA), under development since 1997 Working group: GM, Ford, Chrysler + LSP Target: measurable quality Error categories + Weighting factor per category Sum of points / Word numbers = Quality index Focused primarily on technical texts such as service manuals for vehicles LISA QA Model 3.0 EN-15038 European Quality Standard (four-eyes principle) Tekom quality criteria (working group, 2005) Wir nehmen Qualität beim Wort. 5. Oktober 2009 4
Challenges in global organizations Overall challenges: Global competition shorter product development cycles = need to react faster ( time to market ) & more frequent updates More product variants information maintenance & costs Growing number of foreign languages more resources, higher administrative costs Increasingly complex information technologies CMS, XML, data bases, workflow system Effects on documentation and translations: Chunks of documents = reduced context information, inconsistencies CAT systems = little context MT technologies = linguistic quality, style Complex terminology not always managed Multiple suppliers = varying quality, maintenance of translation memories and terminology Lack of universally accepted quality standards Wir nehmen Qualität beim Wort. 5. Oktober 2009 5
Origin of quality problems Linguistical Quality of the source documentation Semantics e.g. ambiguity Syntax Technical Mode of operation of translation memories Segmentation Fuzzy Matches Organizational Workflow (order of tasks, schedule, chunks of information translated) Selection of translators Communication Wir nehmen Qualität beim Wort. 5. Oktober 2009 6
Translation memories and quality Context is missing Undiscovered errors propagate themselves (as 100% matches) Segmentation issue (incomplete sentences) Fuzzy matches can lead to false translations (confirmed too fast) 99% Wir nehmen Qualität beim Wort. 5. Oktober 2009 7
Today s practice Quality assurance is mostly a manual process Quality assurance not a high priority in many organizations: The translator is in charge, I haven t heard anything negative... Spot checks, uncritical reuse of existing translations (TM) Often restricted to formal aspects such as spelling or layout Large terminologies (5,000+ terms) not really checked Less familiar languages are rarely checked. Key problem: time & costs of a thorough and systematic quality assurance Wir nehmen Qualität beim Wort. 5. Oktober 2009 8
Allocation of translation costs 20% 10% 10% 10% Project Management & Communication Terminology Translation Quality Assurance 50% Technical Preparation and Memory maintenance QA costs = approx. 20% of total translation costs (if compliant with EN 15038 requirements) 50.- per hour and Ø 8,000 words per day = 0.05 per word (approx. 20% variance) Wir nehmen Qualität beim Wort. 5. Oktober 2009 9
Limitations of human QA Problems with large volumes: Time: reviewer 6,000 10,000 words per day Resources: 1,000 pages per day in 30 languages? Problems with the results: Terminology: how to verify 5,000+ terms?! Accuracy of formal checks: numbers, typography, completeness Consistency: work by teams of translators Check of unknown/less known languages Problems with feedback and evaluation: Immediate feedback to translators? Objective + reproducible evaluation of quality? Wir nehmen Qualität beim Wort. 5. Oktober 2009 10
Limitations of automatic QA It only detects formal error patterns (compares strings with preset values) No intelligent checks requiring judgment (e.g. misunderstanding of source text) Level of noise (false error messages) Solution: Automatic QA for formal checks, Human QA for checks requiring intelligence. Wir nehmen Qualität beim Wort. 5. Oktober 2009 11
Which checks can be automated? Quality issue Automatic Manual ErrorSpy since 2003 Completeness Generates automatically list of errors Consistency Numbers Computes automatically a quality index (evaluation of translation) Style Tags Terminology Typography Understanding Wir nehmen Qualität beim Wort. 5. Oktober 2009 12
How can quality be measured? Error categories + Weighting factor per category Sum of error points / Number of words = Quality index ErrorSpy reports use metrics adapted from SAE J2450 Can be used for vendor assessment: Objective evaluation Reproducible assessment of vendors (configuration of checks saved as a template) Wir nehmen Qualität beim Wort. 5. Oktober 2009 13
Example: translation workflow at Siemens Automated workflow since end of 2004 Situation was/is: High volumes Many languages Many suppliers (MLV, SLV) Goal and requirements were: Check linguistically 100% of translation projects Standardize Automate Automatic quality checks (QCs) with instant reports Predefined error levels Actions triggered by report results (Quality gates) One measurable, reproducible, objective quality Pilot project completed in Dec. 2004 and continuously improved since Wir nehmen Qualität beim Wort. 5. Oktober 2009 14
Implementation of automated quality assurance Quality gates 4 Quality gates: Automated quality evaluation triggers actions in translation workflow system (SDL Teamworks) A = Pretranslation SL1 = Translation SL2 = Proofreading SL3 = Verification (in-country review) (Check results from in-country review) Slide from common presentation at the Tekom conference in Wiesbaden, Germany, Autumn 2007; Francois Massion (D.O.G.), Thomas Hecht (Siemens) Wir nehmen Qualität beim Wort. 5. Oktober 2009 15
Automatic corrections Some types of errors are formal and not language dependant Example: Some consistency errors Some number errors Special case: terminology correction of UI terms. Term is invariable and tagged as e.g.: <menuitem>exit</menuitem> Better to correct 10% of errors automatically in a reliable way than to correct 0% Possibility to log and undo the automatic corrections Wir nehmen Qualität beim Wort. 5. Oktober 2009 16
Quality monitoring Ø errors per period The issue is not just the quality of one project Need to develop metrics to measure the efficiency of QA policy at the corporate level. Type of errors Origin of errors Errors per language combination Errors per supplier Average number of errors per period Costs of repairs etc. Allocation of errors according to type and period Single project reports (Database) Wir nehmen Qualität beim Wort. 5. Oktober 2009 17
Outlook More automated and instant QA of large volumes of translations Solution for both human and machine translation Improved learning capability (reduction of noise level) Automatic correction of some errors Prevention of errors (feedback to authors, improvement of source documentation) Worldwide accepted standards & metrics for translation quality Web 2.0 cooperation models (e.g. interaction between author, translator, terminologist and reviewer) Wir nehmen Qualität beim Wort. 5. Oktober 2009 18
Thank you for your attention! Contact: François MASSION Email: francois.massion@dog-gmbh.de Internet: www.dog-gmbh.de Tel.: +49 (0) 7152 / 354 11-10 Wir nehmen Qualität beim Wort. 5. Oktober 2009 19