Detecting future social unrest in unprocessed Twitter data

Transcription

1 Detecting future social unrest in unprocessed Twitter data Emerging Phenomena and Big Data Ryan Compton* Craig Lee Tsai-Ching Lu HRL Laboratories Malibu, California Lalindra De Silva The University of Utah Salt Lake City, UT Michael Macy Cornell University Ithaca, NY Abstract We have implemented a social media data mining system capable of forecasting events related to Latin American social unrest. Our method directly extracts a small number of tweets from publicly-available data on twitter.com, condenses similar tweets into coherent forecasts, and assembles a detailed and easily-interpretable audit trail which allows end users to quickly collect information about an upcoming event. Our system functions by continually applying multiple textual and geographic filters to a large volume of data streaming from twitter.com via the public API as well as a commercial data feed. To be specific, we search the entirety of twitter.com for a few carefully chosen keywords, search within those tweets for mentions of future dates, filter again using various logistic regression classifiers, and finally assign a location to an event by geocoding retweeters. Geocoding is done using our previouslydeveloped in-house geocoding service which, at the time of this writing, can infer the home location for over 62M twitter.com users [1]. Additionally, we identify demographics likely interested in an upcoming event by searching retweeter s recent posts for demographic-specific keywords. I. INTRODUCTION Widespread adoption of social media has made it possible for any individual to rapidly communicate with an audience of thousands [2]. Unlike traditional media, where several difficult time-consuming steps must be carried out prior to publication, information in social media becomes publicly available within a few seconds of its creation. In this work we utilize social media s accelerated publication speed to report on events prior their occurrence, while they are still in their planning stages. Very recently, the speed of publication on twitter.com has motivated its use as a tool for the organization of future civil unrest events [3]. This fact, combined with the public availability of twitter.com data, has motivated our use of it as well. We will demonstrate how we Supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior National Business Center (DoI / NBC) contract number D12PC The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/NBE, or the U.S. Government. filter through this large and dynamic supply of information for messages exhibiting several independent and informative features. Early detection of civil unrest events is valuable for several industrial and government applications. For example, if a port is likely to shut down due to a riot, shipping companies may opt to redirect freight in order to prevent unexpected losses. If a massive protest is planned to happen in front of an embassy, governments may elect to postpone diplomatic visits in order to ensure the safety of their politicians. The value of civil unrest forecasting has recently caught the attention of researchers from a wide variety of disciplines [4] [5] [6] [7]. We use early detection rather than prediction. Our method is based on direct extraction of relevant tweets instead of a physical model describing a large-scale theory of population behavior (e.g. [6] [7]). Our forecasts are based on a small number of highly important tweets and are independent of trends observed in the aggregate of all tweets. To be specific, our detection algorithm can be broken down into the application of several filters which we use to continually monitor streaming data from twitter.com, cf alg. 1. Algorithm 1: Future event detection Input: millions of today s tweets on twitter.com Output: a few dozen tweets relevant to upcoming events t 1 = tweets with text containing specific keywords t 2 = tweets in t 1 with text containing future dates t 3 = tweets in t 2 which have passed through a text classifier t 4 = tweets in t 3 whose retweeters localize within nations of interest or whose text contains mentions of specific locations return t 4 The tweets output by alg. 1 are often few in number and rich in information about upcoming Latin American civil unrest events. We believe that any user of our system will be able to easily interpret each day s results with minimal effort. This paper is organized as follows: section II describes each /13/$ IEEE 56 ISI 2013, June 4-7, 2013, Seattle, Washington, USA

2 step of alg. 1 in detail. section III showcases our user interface and has information about the system s past performance. Finally, section IV discusses future work and concluding remarks. II. METHOD Our goal is to generate forecasts of the form: {population, event_type, date, location, probability} Where population describes the demographic of the event participants (eg education, labor, agriculture), event type gives further detail about the reason for the event (eg employment, housing, economic policies), date is the date the event is forecast to occur on, location is the city where we expect the event to occur, and probability is how likely it is that the event will actually happen. We obtain twitter.com data in two ways: purchasing a commercial realtime data feed consisting of a random sample of 10% of all tweets [8] and continually querying twitter.com s rate-limited public API [9]. A. Keyword searches The first filter a tweet must pass is a simple check for mentions of Latin American civil unrest keywords. The advantage of this filter is that it is possible to apply it to the entirety of twitter.com by using the public search API. Latin American civil unrest keywords tend to be esoteric enough that we can often obtain complete coverage for several keywords each day. We have manually identified a collection of 26 keywords which we believe are of substantial importance and query the API to obtain full coverage of these keywords. A set of 335 words was identified by a domain expert for use in our model. Due to twitter.com rate-limiting, searching the public API for all occurrences of all 335 keyswords is not possible so searches for these 335 keywords are constrained to our commercial 10% data feed. B. Future date searches Simple checks for keyword mentions are poor indicators of tweet content. A quick experiment has shown that, in both English and Spanish, only about 20% of tweets that contain a civil unrest keyword are indeed about civil unrest. Furthermore, it is unclear how to forecast an event date from only tweets with certain keywords. We thus apply a second filter, one for mentions of future dates, to the tweets containing protest keywords. Our future date filter searches first for month names and abbreviations in Spanish and Portuguese and second for numbers less than 31 within three whitespace separated tokens from each other. Thus, an example matching date pattern would be 10 de enero. Four-digit years are rare in tweets, in order to determine the year of the mentioned date we use the year which minimizes the number of days between the mentioned date and the tweet s post time. In our example, if a tweet mentions 10 de enero on we assume the user is talking about as is closer in time to than is. Once we have extracted dates from the text, we assert that the mentioned dates occur after the tweets post time. We assume that tweets mentioning past dates are unlikely to be indicative of future events. When the future date filter is applied the number of tweets is reduced substantially, a quick in-house experiment on tweets containing unrest keywords collected from the twitter.com API on found that only 1512 of these tweets also contained future dates. For each tweet passing this filter we tentatively issue a forecast for the mentioned date. C. Logistic regression classification Co-mentions of keywords with future dates does not guarantee that a particular tweet is indeed about civil unrest. We have developed two classifiers to classify tweets based on their relevance to a civil unrest event. Our first classifier is a standard logistic regression classifier trained on tweets. The features for the classifier were unigrams and bigrams that surpassed a frequency threshold of 3 in the training data. The training data was acquired using three annotators through Amazon Mechanical Turk and they annotated 3000 tweets for their relevance to a civil unrest event (pairwise inter-annotator agreement ranged from 0.68 to 0.74). Our second classifier makes use of recent work we have done establishing that tweets from organizations are roughly three-times more likely to be civil unrest-related than similar tweets from individuals [10]. In order to exploit this concept, we designed an auxiliary classifier that classifies the source user type of a tweet into two categories - organizations and individuals. For this classifier, we make use of an ensemble framework for user type identification based on heuristics, an ngram classifier, and a linguistic classifier. The heuristics were designed to capture two strong cues that are characteristic of organization tweets - 1) they almost always contains a URL and 2) organizational tweets rarely contain replied tweets (tweets beginning mentions). The ngram classifier was based on unigrams and bigrams and the linguistic classifier captures several types of linguistic features that are characteristic of tweets in either category. These three components in the ensemble are then utilized in linear combination using another logistic regression classifier to determine the user type of any given tweet. After we have identified the posting user as individual/organization using this classifier, we adjust the forecast probability accordingly, by incorporating the likelihoods to derive the postrior probability of a tweet being civil unrest-related GIVEN its user type. D. Event geocoding Identification of event locations is central to the goal of this project. We infer the location of an upcoming event by searching the tweet for mentions of cities or monuments from a manually compiled list. In the event that no locations are mentioned in text, we assign a location to an event based on the location of retweeters. 57

3 For each tweet passing the logistic regression filter, we query the twitter.com API for user IDs of all the tweet s retweeters. User IDs are then fed into our geocoder (outlined below) and filtered based on whether or not they reside in Latin America. We note here that this filter is remarkably difficult to pass. Of the 1512 tweets collected in the previous step in our example, only 36 passed the Latin America geocoding filter. We assume that civil unrest events tend to occur in population centers. Our geocoded users are reverse-geocoded into the nearest city with population greater than The most commonly occurring city is used as the location of our event. User geocoding: We identify retweeter locations with our previously developed in-house geocoder [1]. At the time of this writing, our geocoder is able to estimate home locations of 62M twitter.com users with a median error of 11.1km. The distinguishing feature of our geocoder is its ability to infer a user s location based on the locations of that user s online social ties. This is accomplished via a technique similar to label propagation [11] applied to the twitter.com network. Here, bidirectional means that edges in our network are formed when user i user j and user j user i at least once. Briefly, our user geocoder works as follows: We begin by extracting home locations for users based on the number of times they have tweeted with location services turned on. When our 10% sample of twitter.com contains 3 or more tweets from a user within a 15km radius we use the geometric median of those tagged tweets to establish the user s home location. This provides us with home locations for users. Experiments with self-reported locations in profile information and time zone pruning showed negligible benefits [1]. Denote the network by N and define a geometric median function φ : N N by Fig. 1: Example forecast. A march related to Petroleos Mexicanos (Pemex) is planned for March 18 in Mexico City. Our system detected the event on March 5th. The interactive map provides end-users with links to retweeter accounts. { n if n is gps-known φ(n) = argmin n i adj. to n d(n i, x) else x (1) where each node n N stores information about the latitude/longitude for the user associated to that node. The distance function is computed with Vincenty s formulas [12]. The optimization in eq. (1) is known as the l 1 multivariate median or geometric median [13]. Our geocoding algorithm can now be stated concisely. Algorithm 2: Geocoding Initialize: N, gps-known ground truth user locations while not converged do foreach n N do n φ(n) end end return N Empirically, alg. 2 converges, providing us with location estimates for twitter.com users. E. Demographics and event code assignment We condense duplicate forecasts for the same date/location into one forecast by averaging their probabilities. Our domain expert has provided us with lists of terms relevant to several demographics and event types in Latin America. To assign a demographic to each forecast we collect the tweet histories of every retweeter of every tweet associated with a forecast and search our list of terms. The most commonly occurring classes of terms are used to assign our forecast s demographic and event code. III. RESULTS Successful end-user interpretation is important. By approaching this problem from the viewpoint of early detection rather than prediction we can more easily provide an audit trail which can be understood with minimal effort. For each forecast generated we provide the tweets used, the retweeter locations, the keywords matched, and links to all retweeter accounts. (cf fig. 1). 58

4 Fig. 2: Cumulative sum of the number of forecasts generated since The addition of twitter.com API searches happened in late February and led to a notable increase in the amount of forecasts generated each day. Fig. 3: Number of events forecast to happen on each day. There is a significant spike on February 14th, these are mostly false positives. A quick look at the audit trails reveals that these false positives were due to twitter.com user s propensity to protest February 14th. number of events forcast nation 50 Argentina 19 Brazil 39 Chile 53 Colombia 15 Ecuador 9 El Salvador 156 Mexico 6 Paraguay 1 Uruguay 82 Venezuela TABLE I: Number of forecasts generated for each country. Mexico is highly active on twitter.com and receives the most coverage from our system. Our system has been in place since (cf fig. 2), in that time the rate at which forecasts are generated has been steadily increasing as we continue to update our algorithms. Incorporation of the twitter API for complete coverage of a small set of keywords happened in late Febuary 2013 and the number of forecasts generated each day can be seen to increase sharply around this time. Aside from the geocoder, it is possible to implement our system without commercial data feeds, i.e. our results are reproducible from publicly-available data. The amount of events forecast to occur on each day is plotted in fig. 3. There is no obvious pattern. In the future, after substantially more forecasts have been generated, it may be possible to use this time series as part of a prediction-based approach. Assessing the performance of our system is relatively straightforward given the audit trails. Manual examination of 283 forecasts generated by this system revealed 157 forecasts which are indeed about upcoming Latin American civil unrest events and 126 forecasts related to sporting events, other public functions, or simple chatter. IV. CONCLUSION Twitter.com has become a powerful tool for the organization of mass gatherings of all types. However, the shear volume of twitter.com makes it difficult to automatically identify new and valuable information in real time. In this work we have provided a straightforward approach for the detection of upcoming civil unrest events in Latin America based on successive textual and geographic filters. Traditional news media is often assumed to be perfectly accurate and can therefore only report on events once they have occurred. The fact that it is now possible to relax the assumption of perfect accuracy and report on events before their occurrence is remarkable and continued work on this project is already in progress. Immediate future work includes more advanced tweet classification using larger training sets, network, and dictionary-based approaches. We also plan to analyze the links shared in tweets for further information on upcoming events. V. REFERENCES REFERENCES [1] D. Jurgens, Inferring location in online communities based on social relationships, HRL technical report, [2] H. Kwak, C. Lee, H. Park, and S. Moon, What is twitter, a social network or a news media? WWW, [3] J. Skinner, Social media and revolution: The arab spring and the occupy movement as seen through three information studies paradigms, [4] E. Stepanova, The role of information communication technologies in the arab spring, PONARS Eurasia Policy Memo No. 159, [5] F. Chen, J. Arredondo, R. P. Khandpur, C.-T. Lu, D. Mares, D. Gupta, and N. Ramakrishnan, Spatial surrogates to forecast social mobilization and civil unrests, Position Paper in CCC Workshop on From GPS and Virtual Globes to Spatial Computing-2012, [6] D. Braha, Global civil unrest: Contagion, self-organization, and prediction, PLoS One,

5 [7] N. Johnson, S. Carran, J. Botner, K. Fontaine, N. Laxague, P. Nuetzel, J. Turnley, and B. Tivnan, Pattern in escalations in insurgent and terrorist activity, Science, [8] [9] [10] L. D. Silva and E. Riloff, Exploiting the textual content of tweets for user type classification, ICWSM (submitted), [11] X. Zhu and Z. Ghahramani, Learning from labeled and unlabeled data with label propagation, Technical Report CMU-CALD , Carnegie Mellon University, Tech. Rep., [12] [13] Y. Vardi and C.-H. Zhang, The multivariate l1-median and associated data depth, Proceedings of the National Academy of Sciences, vol. 97, no. 4, pp ,