Societal Data Resources and Data Processing Infrastructure

Transcription

1 Societal Data Resources and Data Processing Infrastructure Bruno Martins INESC-ID & Instituto Superior Técnico 1

2 DATASTORM Task on Societal Data Project vision : Build infrastructure for large scale social data analysis and processing, of interest to areas such as the social sciences and economics, to be hosted at the National Foundation for Scientific computing (FCCN). Study of large real social networks Data acquisition, storage and processing Knowledge extraction Understand user activity and behavior Analyze social behavior Entity disambiguation Analyze information diffusion and influence patterns Mining of network communities Deep articulation with horizontal tasks (e.g., H1, H2, H3 and others) 2

3 Challenges in Using Societal Data Several heterogeneous sources of relevant information Traditional sources (small scale datasets) Large sets of statistical series on various areas Scientific data repositories from the social sciences Repositories with geographic and territorial information Particular focus : Social media and news (? big data? ) Articles in traditional media (i.e., online newspapers) and comments to these articles Data on Web archives Data from the social Web (e.g., Twitter, FourSquare, Facebook, etc.) Project vision : Deep (automated) study of societal issues involves integrating these sources, and analyzing the resulting datasets by combining techniques for processing textual information and networks 3

4 Statistical Series and Scientific Data Repositories PORDATA features statistical series about Portugal, Portuguese municipalities and Europe, organized into themes (e.g., population, health, family income and expenditure, education, employment, etc.) dados.gov.pt is a open-data initiative/portal that aggregates and publishes information produced by the Portuguese public administration (e.g., public expenditure, electoral results, etc.) Arquivo Português de Inf. Social (now hosted on RCCAP/FCCN) aggregates information collected about the Portuguese society in the context of academic studies (e.g., surveys and opinion pools) 4

5 News Articles, Coments over News Articles, and Social Media Contents Society is reflected in the media that is presented to us Text mining on news articles and on comments to news articles Twitter and Facebook continue to become a more and more important collective, global media voice, and is thus an important story in itself worthy of scientific analysis Science 30 September 2011: Vol. 333 no

6 Horizontal Tasks from DATASTORM Horizontal Task 1 : Data aquisition and extraction Task Leader : Pavel Calado Data crawling data from social media platforms and from the Web Mining textual contents Horizontal Task 2 : Data representation and query validation Task Leader : Alexandre Francisco Building graph-based representations for the data Data structures and algorithms for efficient storage and processing Horizontal Task 3 : Knowledge Discovery Task Leader : Ana Teresa Freitas Algorithms for knowledge discovery from societal data 6

7 Some Previous Work on the Area Previous work in the area 7

8 Using the Social Web for Analyzing Social Change Online social networks are now a fundamental organizing mechanism in recent country-wide social movements Characterizing the networks associated to these movements Discovering communities and influential individuals Examine the temporal and geospatial evolution of the communication activity Comprehend modern societal dynamics! The Digital Evolution of Occupy Wall Street. PLoS ONE 8(5) The Geospatial Characteristics of a Social Movement Communication Network. PLoS ONE 8(3) Structural and Dynamical Patterns on Online Social Networks: The Spanish May 15th Movement. PLoS ONE 6(8) 8

9 Location, Social Ties, and Human Mobility Using location-based social networks (LBSNs) to investigate human mobility and the relationship between social ties between people and co-occurrences in time and space Predicting which individuals may become friends on social networks, based on visited places Geolocating users with basis on their social links Human mobility follows surprisingly regular patterns Understanding individual human mobility patterns Nature 453, (2008) Friendship and Mobility: User Movement In Location-Based Social Networks Proceedings of the 2010 ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 9

10 Quantifying Happiness and Public Well-Being hedonometer.org is an instrument that measures the happiness of large populations in real time Uses Twitter s gardenhose feed (e.g. about 10% of all messages, 100GB of JSON per day). Messages written in English are assigned a happiness score based on the average happiness score of the words contained within (words manually accessed through crowdsourcing). This measure of happiness correlates very well with traditional surveys of well-being. Temporal Patterns of Happiness and Information in a Global-Scale Social Network: Hedonometrics and Twitter. PLoS ONE, 6, e26752,

11 The Hedonometer and Quantifying Hapiness and Public Well-Being 11

12 Text Driven Forecasting and Predicting the Real World from Text Regression models from sparse/noisy word features Often using methods that promote sparse models (e.g., Lasso regression) Many possible applications Box office results Elections and opinion pools Stock value Investment risk Restaurant menu prices Whitepaper from Noah Smith in 2009 : Text-Driven Forecasting Word Salad: Relating Food Prices and Descriptions Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing and Natural Language Learning 12

13 Interfaces to other vertical tasks - Cultoromics and Epidemics - and the cultural Harvard Using n-grams extracted by Google from books Type in a word or phrase in one of seven languages and see how its usage has been changing throughout the last few centuries Studying epidemics with Twitter Use information embedded in the Twitter stream to: Track rapidly-evolving public concern with respect to H1N1 or swine flu Track and measure actual disease activity. Quantitative analysis of culture using millions of digitized books. Science, 2010 The Use of Twitter to Track Levels of Disease Activity and Public Concern in the U.S. during the Influenza A H1N1 Pandemic. PLoS ONE 6(5): e19467 The Expression of Emotions in 20th Century Books. PLoS ONE 8(3): e

14 Interfaces to other vertical tasks - Cultoromics and Epidemics - and the cultural Harvard Using n-grams extracted by Google from books Type in a word or phrase in one of seven languages and see how its usage has been changing throughout the last few centuries Studying epidemics with Twitter Use information embedded in the Twitter stream to: Track rapidly-evolving public concern with respect to H1N1 or swine flu Track and measure actual disease activity. Quantitative analysis of culture using millions of digitized books. Science, 2010 The Use of Twitter to Track Levels of Disease Activity and Public Concern in the U.S. during the Influenza A H1N1 Pandemic. PLoS ONE 6(5): e19467 The Expression of Emotions in 20th Century Books. PLoS ONE 8(3): e

15 Some National Projects as Well 15

16 The rest of this session at the DATASTORM workshop Silvio Moreira, INESC-ID The REACTION and POPSTAR projects João Vasconcelos, AMA Introducing the dados.gov.pt portal Paula Carvalho, INESC-ID Natural language processing for the social sciences Hopefully, some discussion afterwards 16