Do big data hide or reveal stories? Recording history in the NewsReader project Funded by the EU - FP7 Work Programme Call FP7-ICT-2011-8 Research theme Information and Communication Technologies, challenge 4.4 - Area Intelligent Information Management as project ICT 316404 Piek Vossen Faculteit der Letteren VU Universiteit Amsterdam Seminar on Reference Cultures and Identity formation NIAS, Wassenaar, April 25th 2013
Do big data hide or discover stories? Big data and the financial crisis What is the impact of the economic crisis on the car industry? Lexis Nexis estimates that their archive contains 10 million English news documents on car industry for the last 10 years. 2
3
4
How news accumulates over time Fall of Srebrenica 1995 2000 Lehman Brothers Bankrupcy 2008 2009 Project X Haren 2010 2011 2012 2013? Facebook Direct report Later in time Moslim women and deportation children are loaded genocide Intentions, responsible, strategy, judgment into trucks and transported Moslim men are shot Dijksterhuislezing, 19 april 2013, VU Universiteit Amsterdam Grexit 5
Big data asks for new solutions Search results are too plentiful and rich to browse through result lists, even for faceted browsing Results contain too many duplicates, overlap, repetitions: how do you distinguish old from new information? Every single result is incomplete, only tells you a small part of the story Results are inconsistent and contradict each other News is opiniated and coloured Most information is not authorised: who tells the right story? 6
NewsReader-ICT 316404 Develop technology to automatically process massive streams of daily news from sources in 4 different languages: what happened, where, when and who is involved which temporal and causal relations hold between extracted events and what do they tells us about people's intentions (the why) place the cumulated result in a knowledge store that can handle the dynamic growth of information on events over time, ultimately recording the reported history: a history recorder. Organise and visualise massive amounts of information as stories, scripts, plots to provide more effecient access Partners: Netherlands (VU, LexisNexis, Synerscope), Spain (Basque University), UK (ScraperWiki) and Italy (Federation Bruno Kessler, Trento) Januari 2013 December 2015 7
8
From text to structure 9
http://www.standaard.be/artikel/detail.aspx?artikelid=dmf01072006_034 Part production Polo transferred from Spain to Vorst Wat happened in the Belgium place Vorst in 2006? A spokesman of Volkswagen in Germany commented that it is possible that the production at Volkswagen Vorst will be increased. part of the current production of Golf in Vorst will be transferred to Wolfsburg, without social consequences. 10
Jobs at Vorst April 2006: Juli 2006: Less Golfs produced in Vorst, maybe more Polos. If not, we have a problem, says a union representative...chances that Vorst will not make any Polos next year are minimal, because the factory invested this year in a special new welding installation specific for Polo cars. November 2006: Polo production in Vorst, no jobs lost in Spain but extra jobs in Belgium. Augustus 2006: production of Polo from Spain to Eastern-Europe because of social problems in Pamplona and maybe to Vorst in Belgium Volkswagen stops the production of Golf in Vorst: 3,500 jobs are lost November 2009: Audi plant in Vorst stops the production of Polo: 300 jobs lost 11
Grounded Annotation Framework Sources report on events mentions of events Sources: texts, databases on payment transactions, social-economic data, telephone usage, internet usage, etc, sensors such as cameras. Most events take place in reality and not in the source instances of events defined by a URI Linked Open Data (Semantic Web) GAF links all mentions of the same event to a unique instance URI and combines all information from each mention New information from mentions in future sources (after the event date) is continuously merged with the information around that single instance: reinterpretation through historical coreference Published in NAACL-2013 workshop (Fokkens et al.) 12
Bron: Bron: http://www.autoblog.nl/archive/2006/11/21/volkswa gen-fabriek-belgie-moet-stoppen-met-de-golf http://www.autointernationaal.nl/artikel.p hp?id=2868 Tijd: 21-11-2006 om 15:07 Tekst: Volkswagen fabriek België moet stoppen met de Golf. stop with the Golf. Er gingen namelijk geruchten over zware saneringen, van 1.000 tot 4.000 ontslagen. disappear sure 3.500 jobs Action: Participants: Time: Location: There were rumours about heavy cuts, of 1.000 up to 4.000 fired Bron: Tijd: 21-11-2006 11:26 Tekst: Bij de Volkswagen fabriek in Vorst verdwijnen zeker 3.500 banen jobs gone in Vorst Plant http://www.standaard.be/kanaal/index.aspx? kanaalid=261&pageid=15 loses Golf Tijd: Di 21/11/2006 Tekst: Tot 4.000 banen weg in Vorst Fabriek verliest Golf BRUSSEL - Boze arbeiders blokkeren de poorten van de Volkswagenfabriek in Vorst, nadat bekend raakte dat zeker 3.500 van de 5.300 banen verdwijnen. sure 3500 of the 5300 jobs disappear DECREASE? NR OF JOBS, VW PLANT 21 11-2006 Vorst, Belgium Database unemployment figures Bron: http://nl.wikipedia.org/wiki/audi_brussel Tijd: 13 mrt 2013 om 13:24 Tekst: Op 21 november 2006 kwam het nieuws dat Volkswagen Vorst de productie van de Golf verliest en er meer dan 3500 van de 5000 banen geschrapt zouden worden. production of the Golf lost and there more than 3500 of the 5000 jobs canceled should be 13
How to determine mentions WORD SEQUENCE. Bij de Volkswagen fabriek in Vorst verdwijnen zeker 3.500 banen... CONCEPT SEQUENCE....PLANT...DISAPPEAR...3.500..JOBS RELATIONS & IMPLICATIONS TIJD, PLACE (DISAPPEAR, 3.500 JOBS) (OWNS, VOLKSWAGEN, PLANT) (OPERATE, PLANT, JOBS) PROVENANCE autointernationaal SOURCE Language Technology makes STATEMENT CERTAIN 14
What has been said about 2006? VOLKSWAGEN VORST VORST, BELGIUM VOLKSWAGEN LOCATED-IN OWNS HAS-NAME autoblog.nl autointernationaal.nl standaard.be nl.wikipedia.org 73000 annual GOLF 120000 annual AUDI 180000 annual PRODUCES PLANT November 2006 POLO PRODUCES PRODUCES 1000 3500 4000 FUTURE (DISAPPEAR) JOBS HAS JOBS 5300 PATIENT 5000 DISAPPEAR November 2006 March 2013 15
5 years later: 2011 AUDI BRUSSEL VORST, BELGIUM HAS-NAME VOLKSWAGEN LOCATED-IN OWNS nl.wikipedia.org PLANT PRODUCES HAS HAS March 2013 AUDI ROBOTS 450 JOBS 2372 120000 annual 2011 16
From structure to stories What to do with billions of events from millions of news articles? 17
Plots Human motivation, intentions, responsible (Bremond 1980, Brooks 1992, Ryan 1991): become market leader increase profit by transferring production to Eastern Europe Dramatic effect: pressure rises: Porsche buys shares in Volkswagen climax: speculation on takeover of Volkswagen resolution: Volkswagen takes over Porsche Wiedeking sued by investment companies 18
19
Scenario-1: Connecting co-occurring clusters Pyongyang (25 milion hits on Google) Mahmoud Ahmadinejad (21 million hits on Google) 2012 2013 20
Scenario-1: Connecting co-occurring clusters How to find the needle in the haystack that connects two trendy topics? http://www.couriermail.com.au/news/breaking-news/n-korean-official-meets-with-ahmadinejad/story-e6freoo6-1226463500114 21
Scenario-1: Connecting co-occurring clusters 22
Scenario-2 Connecting disjoint cluster Ontslagen na overname Ontslagen na overname management transfer 2005 2013 23
Scenario-3 Cumulation to a critical point - unemployment - price of wheat - oil price - production of cars - market shares - measurements with impact - people, organisations that may have impact 2005 2013 24
Determining the value of a connection impact of participants trendy: persons that frequently occur in the news, e.g. Kim Young-Nam strength of opinions on participants: sentiment analyse role and function (decision maker with power) past: participants with a 'backpack', involved in a previous event with impact impact of events impact potential of the participants of an event: everything Obama does... trendy: frequency in news strength of opinions on events type of event (disasters with impact, fraude, corruption, bankrupcies) critical states of important factors, e.g oil price, price of wheat, market shares, monopolies 25
Impact Anything done by an important person is potentially important: Barack Obama has a cold. If an event has impacts then all its participants are potentially important: Belgium and Dutch boys that fight in Syria 26
Searching for the why? The internet has a wealth of opinions on possible reasons behind and explanations of events: conspiracy theories opinions of ordinary citizins views and opinions of research journalists and scientist 27
Dijksterhuislezing, 19 april 2013, VU Universiteit Amsterdam 28
as reason for Crisis used as reason for massive firing of people Dijksterhuislezing, 19 april 2013, VU Universiteit Amsterdam 29
A view of the past, gives... 30
\ Visionairs 31