PoliticalMashup. Make implicit structure and information explicit. Content

Transcription

1 1 2 Content Connecting promises and actions of politicians and how the society reacts on them Maarten Marx Universiteit van Amsterdam Overview project Zooming in on one cultural heritage dataset A few example applications Research ideas for NLP-scientists. Catch Meeting HiTiMe Who am I? 4 project Political scientist turned computer scientist My field: Theory of XML Database Systems Semi Structured Information Retrieval Cooperation with Tweede Kamer Koninklijke Bibliotheek, historians at NIOD, DNPP Large scale data integration project 2 years NWO funded infrastructure project Partners: U. Amsterdam, Groningen and Tilburg Ongoing with irregular funding since Goal of 6 Mashup of what and how? Making huge amounts of textual data available for large scale automatic quantitative data and content analysis done by scientists from the humanities and social sciences. 4 data sources Promises and actions of politicians Reactions on those in media and general public Connect data on Political entities Time Topics 7 Data sources 8 Used techniques Promises Election manifestos, mostly scans, DNPP Party websites and blogs, Archipol Twitter of politicians Actions Parliamentary proceedings, mostly scans, KB Reactions News media User generated content Fora, Blogs, Comments on news, Twitter Text analytics and XML DB and IR technology Named entity recognition and normalization Data mining, Machine Learning, hand-crafted rules Natural Language Processing, Language Models Make implicit structure and information explicit.

2 9 Zoom in on one data corpus 10 Longitudinal data weakly measurement for over 150 years very stable measurement procedure and data model 11 Data about human behaviour 12 Often rather boring 13 But sometimes full of drama and excitement 14 Loads of measurement points days, topics, 7.5 miljoen speeches 15 Digitally available De Handelingen der Staten Generaal (Dutch Hansards) 16

3 17 About this collection 18 Same data: different views very sparse available metadata very rich metadata sits hidden inside the raw data Rich data model Meeting (1 Day) Topic Stage direction Scene Stage direction Speech Paragraph Raw data in PDF XML styled with stylesheet Machine readable XML format 19 Some applications of this 20 Content and structure search Combine IR style keyword search with restrictions on structure. E.g., return speeches by Wilders about Islam 21 Exhaustive data collection 22 Link the proceedings to entities Example query for NIOD historians Search for paragraphs about fascisme OR nazisme OR dictatuur OR (nazi AND dictatuur) OR... Return a tsv file with for each hit date speakername speakerid speaker-party... NIOD query Who is speaking? Who says what to whom? Applications Summary of one speaker On old OCRed data: Linking and resolving entities 23 Application: Interruption graph (Attackogram) 24 NLP research topics MP A interrupts B A speaks during the block of B.

4 25 0) Topics Common European thesaurus ) Populist language in parliament PhD Thesis Jan Jagers (2006). detection classification (sentence, paragraph, speech level) 27 2) Automatically detecting promises ( toezegging ) by ministers in Parliament https: //zoek.officielebekendmakingen.nl/kst pdf (pagina 56) Eerste Kamer has a nice database online 28 Example De voorzitter: Ik constateer dat wij bijna aan het einde van deze vergadering zijn gekomen. Wij hebben nog tijd om even de toezeggingen langs te lopen. Ik vraag iedereen om op te letten of er niets over het hoofd is gezien. Ik zal dit snel doen en daarna spreken wij nog even over het vervolg. De toezeggingen. Na de zomer ligt het wetsvoorstel bij de Kamer. Er komt een brief om de Kamer erover te informeren op welke wijze er voorkomen wordt dat er expertise verloren gaat. Minister Van Bijsterveldt-Vliegenthart: Dat heb ik niet toegezegd. Beslist niet. Nee, dat doe ik niet, want ik heb dat niet toegezegd. 29 3) Opinion detection 30 4) Detect type of speech Detect opinions expressed about entities and topics. (Speaker is known) Detect reported speech. Interruption, attack, answer, speech ( betoog ), stage-direction,... h-ek tijdslijn.html 31 5) Detect bullshit 32 6) Spelling normalization Tautologiëen... Regels zijn regels, Op is op p p Dutch had many spelling reforms. Leads to lower recall. Search in new spelling, return results in old spellings. het is wat het is

5 33 Lots of data available: happy to share 34 Thanks Now: 15 years of Dutch Parliamentary Proceedings in rich XML Now: 200 years more in poorer XML, slowly getting richer. Parliamentary proceedings from EU (15y), UK (75y), Spain (40y), Scandinavian countries,... Election manifestos (provincial elections 2007 and 2011) All tweets, blogs, Flickr and Youtube of all Dutch national politicians since 1.5 year.