Towards automatic terminology extraction for Norwegian based on parallel corpora

Size: px

Start display at page:

Download "Towards automatic terminology extraction for Norwegian based on parallel corpora"

Tobias Watkins
8 years ago
Views:

1 Towards automatic terminology extraction for Norwegian based on parallel corpora Gisle Andersen LSP Conference, Vienna 8 July 2015

Background and contents NHH is developing a national infrastructure that integrates terminological language resources Termportalen; WP7 in CLARINO project (NFR) Many

2 Background and contents NHH is developing a national infrastructure that integrates terminological language resources Termportalen; WP7 in CLARINO project (NFR) Many specialist fields lacking systematic terminology Case: Sjøfartsdirektoratet (Norwegian Maritime Authority) Contents Introduction: aim Data and methods Pattern matching Conclusion 2

3 Aim of work Purpose: providing aid to field experts where systematic terminology work is lacking A generic system meant to enhance terminology for various domains Maximising value of existing tools and language resources Setting up an infrastructure, a production line for term extraction (TE) Using a variety of techniques for parallel corpusbased TE 3

4 The necessary disclaimer What is extracted through computational methods are always term candidates. Need for subsequent manual check by field experts Need to supply additional information about concepts (definitions, structure, term variation); cf. e.g. Heylen & De Hertog (2015) 4

5 Towards automatic terminology extraction for Norwegian DATA AND METHODS 5

Current version: translated regulations from International Maritime Organization (IMO) - small;

6 The corpus (1/2) Sjøfartsdirektoratet (NMA) Parallel corpus of translated texts (EN NO) Policies and legislation relating to shipping - navigation, communication, safety, etc. Current version: translated regulations from International Maritime Organization (IMO) - small; currently 9 items To be extended to include - Skipssikkerhetsloven / The Ship Safety and Security Act - NMA s own regulations - Etc. 6

7 The corpus (2/2) Title of regulation Navn på forskrift TCA2 fil TCA2 fil Regulations of 1 July 2014 No on the Forskrift 01. juli 2014 om bygging av RCS_E RCS_N construction of ships skip Regulations of 1 July 2014 No. 944 Forskrift 1. juli 2014 om farlig last på RDG_E RDG_N on dangerous goods on Norwegian ships norske skip Regulations of 1 July 2014 No on fire Forskrift 1. juli 2014 om brannsikring RFP_E RFP_N protection on ships på skip Regulations of 1 July 2014 on life-saving Forskrift 1. juli 2014 om RLS_E RLS_N appliances on ships redningsredskaper på skip Regulations of 5 June 2014 No. 805 on medical examination of employees on Norwegian ships and mobile offshore units Forskrift nr. xxxx om helseundersøkelse av arbeidstakere på norske skip og flyttbare RME_E RME_N Regulations of 5 September 2014 No on navigation and navigational aids for ships and mobile offshore units Regulations of 1 July 2014 No. 955 concerning radiocommunication equipment for Norwegian ships and mobile offshore units Regulations of 5 January 2014 No on a safety management system for Norwegian ships and mobile offshore units IMO standard marine communication phrases (SMCPs) innretninger Forskrift om navigasjon og navigasjonshjelpemidler for skip og flyttbare innretninger Forskrift 1. juli 2014 om radiokommunikasjonsutstyr for norske skip og flyttbare innretninger Forskrift om sikkerhetsstyringssystem for norske skip, og flyttbare innretninger IMOs standarduttrykk for maritim kommunikasjon RNN_E RRR_E RSM_E SMCP_E RNN_N RRE_N RSM_N SMCP_N 7

8 System architecture for TE 8

9 Step 1: Text conversion: doc html xml 9

10 Step 2: Alignment of parallel corpus texts 10

11 Step 3A: Pattern matching Premise: recognisable patterns in sentence and paragraph structure, punctuation, etc. suggesting termhood Extraction based on regular expressions (perl) <s>b) barges;</s> <s>the spooling device shall:</s> <s>a) initial certification upon changes in use;</s> <s>e) handrails, corridors and passageways, doorways, doors, lifts, vehicle decks, passenger lounges, accommodation and washrooms shall be Wire/chain stoppers shall be dimensioned for a safe working load <s>d) lektere</s> <s>spoleapparatet skal:</s> <s>a) førstegangssertifisering ved endret bruk</s> <s>e) Håndlister, korridorer og ganger, døråpninger, dører, heiser, bildekk, passasjersalonger, innredning og toaletter skal være En wire- og kjettingstopper skal være dimensjonert for en sikker arbeidsbelastning 11

Step 3B: Check of terminological inventory Premise: if word/sequence of words is already registered as term in other component of Termportalen, it has high termhood (it is likely to constitute

12 Step 3B: Check of terminological inventory Premise: if word/sequence of words is already registered as term in other component of Termportalen, it has high termhood (it is likely to constitute a term in current context also) Question 1: same or different translation relation Question 2: same or different domain Methodological issue: inflected forms in texts; base form in term base 12

13 Step 3C: Neology detection Premise: if word/sequence of words can be shown to be a neologism (domain-specific vocabulary), it has high termhood (is likely to be a term) Check against inventory of words in large general language corpus (GLC); Norsk aviskorpus (Norwegian Newspaper Corpus, NNC; cf. Andersen 2012; Andersen & Hofland 2012) Check among neologisms registered in NNC s neology database 13

14 Step 3D: Monolingual/bilingual lexicon lookup Premise: if word/sequence of words is found among the lexical inventory in a mono/bilingual technical or specialised dictionary, it has high termhood Agreement with Kunnskapsforlaget to reuse some of their manuscripts 14

15 Step 3E: Association measures (AMs) Premise: terms are often constituted as collocations, i.e. words with a strong tendency to co-occur, so strong collocations may be seen as indicators of termhood Association measures, statistical measures of unithood/termhood (Heylen & De Hertog 2015) Important to select adequate AM for TE, e.g. Pointwise Mutual Information, Chi-square (cf. Lyse & Andersen 2012) Collocation patterns should be compared with GLC data (NNC) 15

16 Step 3F: Parsing techniques Premise: terminological units are typically constituted as (complex) noun phrases; output from syntactic parsing may give good guidance towards terminological units Parsers for Norwegian and English: INESS project (UiB; cf. Rosén 2012) 16

17 Towards automatic terminology extraction for Norwegian A CLOSER LOOK AT PATTERN MATCHING 17

18 Term extraction based on pattern matching final loading conditions Hydrostatics containing the following parameters as a function of the draught with a specified reference point Endelige lastetilstander Hydrostatikk som inneholder følgende parametere som funksjon av dypgang med spesifisert referansepunkt?? displacement deplasement KB KB centre of buoyancy oppdriftssenter If warranted by the ferry's size or type Når fergens størrelse eller type tilsier det the Norwegian Maritime Authority may require the mooring arrangement to be dimensioned for a mooring force higher than 30 tonnes kan Sjøfartsdirektoratet kreve at fortøyningsarrangementet blir dimensjonert for høyere fortøyningskraft enn 30 vekttonn KM transverse metacentre above the baseline KM tverrskips metasenter over basis AwT waterline area TP1 tonnes per unit submersion MT1 moment to change trim LCF longitudinal centre of flotation LCB longitudinal centre of buoyancy AwT vannlinjeareal TP1 enhets neddykking MT1 enhets trimmoment LCF langskips flotasjonssenter LCB langskips oppdriftssenter 18

19 Output of procedure: term database file (tsv) 19

20 The next stage: manual editing in Termportalen 20

21 Towards automatic terminology extraction for Norwegian CONCLUSION 21

22 Other remaining tasks The output of each processing procedure: a bilingual list of term candidates Precision and recall needs to be checked against a gold standard Will be developed via manual term extraction performed by field experts/research assistant The performance/contribution of each module will be checked separately Degree of overlap needs alto to be checked 22

23 Summary A hybrid approach, using a combination linguistic and statistical approaches to (bilingual) TE combining the strengths of both approaches at the same time attempting to utilise and maximise the value of old existing language resources although based on data drawn specifically from the maritime sector, the production line and infrastructure proposed here is meant to be generic and applicable in (all) other domains 23

24 References Andersen, Gisle, ed Exploring Newspaper Language - Using the web to create and investgate a large corpus of modern Norwegian. Amsterdam: John Benjamins. Andersen, Gisle, and Knut Hofland Building a large monitor corpus based on newspapers on the web. In Exploring Newspaper Language - Using the web to create and investigate a large corpus of modern Norwegian, edited by G. Andersen. Amsterdam: John Benjamins. Heylen, Kris, and Dirk De Hertog Automatic term extraction. In Handbook of Terminology, edited by H. J. Kockaert and F. Steurs. Amsterdam: John Benjamins. Lyse, Gunn Inger, and Gisle Andersen Collocations and statistical analysis of n-grams. In Exploring Newspaper Language - Using the web to create and investigate a large corpus of modern Norwegian, edited by G. Andersen. Amsterdam: John Benjamins. Rosén, Victoria Exploring corpora through syntactic annotation. In Exploring Newspaper Language - Using the web to create and investigate a large corpus of modern Norwegian, edited by G. Andersen: John Benjamins. 24

Overview of MT techniques. Malek Boualem (FT)

Overview of MT techniques. Malek Boualem (FT) Overview of MT techniques Malek Boualem (FT) This section presents an standard overview of general aspects related to machine translation with a description of different techniques: bilingual, transfer,