Joint Research Centre

Transcription

1 Joint Research Centre Open Source Monitoring Tools and Applications emm.newsbrief.eu Serving society Stimulating innovation Supporting legislation

2 Open Source Monitoring - Overview EMM Introduction Custom Domains Processing Features End User Applications Collaboration Spotting 2

3 EMM Architecture 2

4 (Definition) Social Media* WWW Blogs

5 (Definition) Sources Input 4000 News Sites articles per day Languages 70 Categories Classes 1000 classes keywords Social Media* Blogs WWW Runs 24/7 Visitors Developed, Built and Maintained by JRC

6 EMM Categories Powerful classification engine Based on user defined keywords/patterns Allows boolean combinations, proximity and wildcards Support for Arabic and similar (automatic pronoun prefixing) Support for chinese and similar (no whitespace) Categories can be overlapping, no ontology, Multilingual Categories defined for: Countries Themes, EC- Institutions and Agencies, Policy Areas, Commissioners, Diseases and many many more 5

7 Example of 1 category

8

9 Automated Entity Extraction persons and organizations based on continuously updated list of entities, many language specific synonyms.

10 Automated Entity Extraction persons and organizations based on continuously updated list of entities, many language specific synonyms. Quote Extraction Supported Languages:ar,bg,da,de,en,es,et,fr,it,nl,no,pl,pt,ro,ru,sl,sv,sw,tr

11 Automated Entity Extraction Geo Tagger Multi-lingual gazetteer of over1.5 million entries (growing) persons and organizations based on continuously updated list of entities, many language specific synonyms. Quote Extraction Supported Languages:ar,bg,da,de,en,es,et,fr,it,nl,no,pl,pt,ro,ru,sl,sv,sw,tr

12 Automated Entity Extraction Geo Tagger Multi-lingual gazetteer of over1.5 million entries (growing) persons and organizations based on continuously updated list of entities, many language specific synonyms. Quote Extraction Event Extraction Supported Languages:ar,bg,da,de,en,es,et,fr,it,nl,no,pl,pt,ro,ru,sl,sv,sw,tr

13 Automated Entity Extraction Geo Tagger Powerful Categorisation Engine (a.k.a. Alerts) Tonality Sentiment Detection Duplicate Detection persons and organizations based on continuously updated list of entities, many language specific synonyms. Quote Extraction Automatic Language Detection Meta-Data Filtering Clustering and Story Tracking Alerting System (SMS - ) Index of all text and metadata (Search) Multi-lingual gazetteer of over1.5 million entries (growing) Statistical Analyser RSS/KML Services for all extracted information Event Extraction In-line Statistical machine Translation Multi-document summarization. Supported Languages:ar,bg,da,de,en,es,et,fr,it,nl,no,pl,pt,ro,ru,sl,sv,sw,tr

14 Web Interface HTML/KML/RSS Index Search SMS/ Notifications Situation Reports/Bulletins

15 Web Interface HTML/KML/RSS Index Search SMS/ Notifications Situation Reports/Bulletins

16 Web Interface HTML/KML/RSS Index Search SMS/ Notifications Situation Reports/Bulletins

17 Web Interface HTML/KML/RSS Index Search SMS/ Notifications Situation Reports/Bulletins

18 Web Interface HTML/KML/RSS Index Search SMS/ Notifications Situation Reports/Bulletins

19 Web Interface HTML/KML/RSS Index Search SMS/ Notifications Situation Reports/Bulletins

20 Web Interface HTML/KML/RSS Index Search SMS/ Notifications Situation Reports/Bulletins

21

22 EMM OSINT Suite Desktop software frontline toolkit for analysts in law enforcement - extending for TTO use. Exploits EMM Technology Tools For typical OSInt Process Google/Bing searches automated result caching WebSite Crawling Document import and analysis (PDF, WORD) Database import Tools for drilling down the extracted data Easy to download and install, use wiki is here: wiki.emm4u.eu gerhard.wagner@jrc.ec.europa.eu Acquire Documents Extract Information Analyse Organise 10

23

24

25

26

27 Ontopopulis: Automated Category Learning a weakly-supervised multi-lingual system for statistical knowledge-poor leaning of semantic classes and co-occurring terms inputs: set of words categorized in different semantic classes, and unannotated text corpus The system learns typical contexts for each of the input classes and then learns additional terms from these classes and also cooccurring terms Ontopopulis uses vector space models to present each input term and category via a vector of its typical contexts 15

28 Ontopopulis Architecture Text corpus Seed: train bus truck car Extraction of contextual features Contextual features: driver of the X : 2.6 X plowed : 2.2 X was parked : 2.2 stopped a X : 2.2 collided with another X : 2.1 Stop words New term extraction New terms: vehicle van lorry taxi minibus

29 Experiment with Collaboration Spotting Project Pixel Detectors Medipix Timepix OSInt Pilatus Google Patent Archive Corpus Corpus 200 Corpus 200 Corpus docs 200 docs 200 docs docs Ontopopulis Categories

30 Some newly learned terms for pixel photon counting pixel detector hybrid pixel detectors atlas counting medipix cms neutron cmos pixel cdte asic photon hybrid semiconductor silicon pixel dectris readout chip readout ray hybrid silicon silicon ccd cmos pixel prototype gamma ray pilatus detector gaas scintillation position sensitive modern This experiment was carried out with the Collaboration Spotting project at CERN The class expansion algorithm learned new candidate terms. We evaluated the top-scored 60 and found that about 90% of them are relevant terms, representing other types of pixel detectors or at least they are strongly related to the domain photo pixilated

31 Thank You emm.newsbrief.eu wiki.emm4u.eu 19