Automated Multilingual Text Analysis in the Europe Media Monitor (EMM) Ralf Steinberger. European Commission Joint Research Centre (JRC)

Transcription

1 Automated Multilingual Text Analysis in the Europe Media Monitor (EMM) Ralf Steinberger European Commission Joint Research Centre (JRC) EMM: LT-Accelerate, Brussels, 24 November 2015 Agenda JRC: Who we are what we do our customers. Europe Media Monitor (EMM) family of applications Importance of multilingual media analysis Visualisation of trends and data distributions Free multilingual Language Technology resources

2 JRC - Who we are European Commission (scientific-technical arm of public administration) Non-commercial Multi-disciplinary / multilingual EMM facts Europe Media Monitor (EMM) news gathering - A few facts ~ 220,000 articles / day in over 70 languages ~ 5000 Sources (world-wide, with focus on Europe) news sources (web portals) specialist medical sites ~ 20 commercial newswires 24/7, updated every 10 minutes 13 languages translated into English (SMT) Complemented with social media analysis Articles are fed into the various EMM applications:

3 EMM users EMM media monitoring users wide coverage, world-wide European Commission (most DGs) and other EU Institutions EU Agencies: e.g. Public Health (ECDC), Food Safety (EFSA), Chemicals Bureau (ECHA), etc. EU Member State organisations: e.g. Public Health, law enforcement authorities, parliaments, crisis management/humanitarian International and extra-european organisations: e.g. various UN organisations Centres for Disease Prevention and Control in the US, Canada, China, The public: ~30,000 anonymous internet users of publicly accessible EMM systems. Combined between 1 and 2 Million hits per day EMM-NewsExplorer Link news over time and across languages 21 languages Daily clustering Long-term monitoring Name recognition and spelling variant matching Cross-lingual news linking

4 EMM-NewsBrief Latest news, categories Over 70 languages Updated every ten minutes Over 1,000 news categories Most used EMM application for real-time media monitoring Medical Information System MedISys

5 EMM-NewsDesk Human moderation NewsDesk allows semi-automatic production of PDF in-house newspapers. Input: NewsBrief Widely used. EMM Public free access EMM output (incl. RSS, KML, Maps) is publicly accessible at Except some functionality (e.g. mynews, NewsDesk). Apps on Google Play: Apps on itunes:

6 Agenda JRC: Who we are what we do our customers. Europe Media Monitor (EMM) family of applications Importance of multilingual media analysis Visualisation of trends and data distributions Free multilingual Language Technology resources Importance of multilingual monitoring News content is complementary across languages and countries, not only regarding contents, but also regarding opinion. News cluster map showing the locations of the news events (link to live map):

7 EMM Multilinguality Complementary coverage of medical news in various languages Italian - German English - French Spanish - Portuguese EMM Multilinguality How to achieve it? Our working conditions: We work with 20 or more languages We do not have access to linguistic analysis tools (parsers, POS-taggers, morphological analysers, dictionaries, ) Our applications need to work more or less the same way for all languages. Our guidelines: Use language-independent rules, if possible. Use as little language-specific resources as possible. Modularity: keep language-specific resources outside the rules plug in any new language by providing the resources for that language. Do not use language pair-specific resources. NewsExplorer covers 21 languages, 210 language pairs.

8 Agenda JRC: Who we are what we do our customers. Europe Media Monitor (EMM) family of applications Importance of multilingual media analysis Visualisation of trends and data distributions Free multilingual Language Technology resources EMM Meta-Information Usable for Visualisation Types of meta-information available in EMM (per article): Time stamp (time of publication) URL News provenance (country) Country the news is about News source Publication language Media type (press, social media, governmental site, ) News category / subject domain Number of related articles (same day or previously) Related articles in other languages Names of persons, organisations and locations (disambiguated) Quotations by and about people Sentiment Any combinations of features; average values; change over time; etc.

9 Trends and Distributions Pies and Bar Charts Language distribution of all European news articles (right) compared to Language distribution for European articles on Science & Technology (left) English and Polish talk more about S&T Trends and Distributions Pies and Bar Charts

10 Trends and Distributions Interactive zoomable maps Trends and Distributions Interactive zoomable maps Event extraction: who (perpetrator) did what (event type) to whom (victims), where (location) and when (time)?

11 Trends and Distributions Interactive zoomable maps Event extraction: map showing events in all languages Trends and Distributions Interactive trend graphs News cluster size NewsBrief (short-term) NewsExplorer (long-term)

12 Trends and Distributions Interactive trend graphs Sentiment over time in English and French articles towards Financial Crisis Trends and Distributions Early warning graphs Medical Information System MedISys Biggest threats to Public Health (country-category combination) Comparing last 24 hours with 2-week average Language-independent Interactive Allowing drilling down

13 Trends and Distributions Social network node graphs Top: Co-occurrence relations (normalised) Right: Who mentions whom in quotes (most quoted people are most central) EMM NewsExplorer Name variants (found in 16h!) live

14 Agenda JRC: Who we are what we do our customers. Europe Media Monitor (EMM) family of applications Importance of multilingual media analysis Visualisation of trends and data distributions Free multilingual Language Technology resources JRC s multilingual linguistic resources Parallel corpora (mostly sentence-aligned) JRC-Acquis (2006): 1 billion word parallel corpus in 22 languages DGT-Acquis (2012): 1 billion word parallel corpus in 22 languages DCEP (2014) billion word parallel corpus in 23 languages Translation Memories (TM) DGT-TM (yearly since 2007): TM in 24 languages; up to 4 million segments per language. ECDC-TM (2012): TM in 25 languages; 32K segments; Public Health. EAC-TM (2013): TM in 26 languages; 78K segments; Education & Culture. Document classification software JEX (JRC Eurovoc Indexer) (2012): software to automatically label texts according to the thousands of categories of the Eurovoc thesaurus; 22 languages. Multilingual Name Variant Resource JRC-Names: 630k Names and their spelling variants All available for download from:

15 Summary: Key points Overview of the Europe Media Monitor (EMM), especially show the visualisation of analysis results; Importance of multilinguality (information complementarity across languages); Machines (and humans) make mistakes: drill-down functionality important.