How To Mine Location Based Social Network Data From Social Networks

Transcription

1 Smart Cities, Urban Sensing and Big Data: Mining Geo-location in Social Networks Daniele Sacco 1, Gianmario Motta 2, Linlin You 3, Nicola Bertolazzo 4, Chen Chen 5 Università di Pavia Via Ferrata 1, 27100, Pavia (PV) 1 [email protected] 2 [email protected] 3 [email protected] 4 [email protected] 5 [email protected] Location based social networks offer spatio-temporal information which can be accessed through public Application Programming Interfaces (APIs) and drew the interest of researchers with diverse scientific backgrounds. This availability of data enables a potential use of geolocated content as an additional, low cost and infrastructureless source of information for urban sensing in Smart Cities. All these aspects bounded with the need of real-time analytics for urban sensing takes to Big Data management and its related issues. A systematic literature review outlines related works and gaps in current research. We propose a reference model to exploit Big Data and Open Data for urban sensing and we validate it by a case study. Finally, we give recommendations for future research about location and mobility mining of social network data. Keywords: big data, urban sensing, smart city, locationbased social network, twitter, data mining, open data 1. Introduction Last years have seen a paradigm shift of the role of the user on the web, from consumer to producer of information. Due to last technological developments and their world-wide dissemination, Web 2.0 changed users approach to information creation and exploitation. The paradigm shift has been also a social shift. Web has become an essential need for people, a sort of commodity, that is used to communicate, interact, share information and even maintain relationships thanks to social networks. The last advent of smartphones, equipped with GPS sensors allowing users to geo-locate themselves, can take to the next shift, from a social and collaborative Web 2.0 to a local and mobile Web 3.0. One among the first achievements has been the integration of Geographic Information Systems Congresso Nazionale AICA 2013

2 Congresso Nazionale AICA 2013 (GISs) and social networks resulting in new location-based capabilities. Social networks that include location information into their contents are called Location Based Social Networks (LBSNs). So, LBSNs offer spatio-temporal information which can be accessed through public Application Programming Interfaces (APIs) and draw the interest of researchers with diverse scientific backgrounds. This availability of data enables a potential use of geo-located content as an additional, low cost and infrastructure-less source of information for urban sensing in Smart Cities. All these aspects bounded with the need of real-time analytics for urban sensing takes to Big Data management and its related issues. Real-time urban sensing use citizens as active and passive sensors and can reveal important insights of human behavior in the city. Diverse use scenarios can enable new perspectives in society level, e.g. community healthcare, public safety, city resource management and transportation management. 2. Systematic Literature Review Systematic Literature Review (SLR) can be regarded as explicitly formulated, reproducible and up-to-date summary [Egger et al, 2008] that includes and extends the statistical results of a meta-analysis methodology. As opposed to narrative reviews, it is based on a structured method that is always explicitly specified at the beginning of the review. Our objective is to identify initiatives, experiences and viewpoints on location and mobility mining of social network data. So, our research question is How to exploit geo-located data from social networks and what level of maturity has reached its application?. So, the expected outcomes of our SLR are: (a) a complete overview of the state of the art, (b) the identification of gaps in current research, solutions, trends and future research and suggestions to the community of researchers and practitioners, and (c) recommendations about best practices for location and mobility mining of social network data. Extracted information contains techniques, issues, models and any other kind of topic useful to provide an accurate snapshot of the current state of location and mobility mining of social network data. 87 out of 109 articles have been selected and classified by year of publication, geographical area, research method and publication channel. There are no significant articles before 2009 because Location Based Social Networks (LBSNs) raised in the same year. Afterwards researchers could start to consider data provided by LBSNs. The articles considered in our research range from 2009 to second quarter of Considering that publications in 2013 refer only to the first months of the year, the number of publications tends to have an exponential growth (2 x ): 2 publications in 2009, 4 in 2010, 14 in 2011, 38 in Case studies represent the majority of the study types. The number of case studies and instrument development publications (87.3% in total) reflects the experimental approach to the issue. It is also motivated by the low number of theoretical papers (5.7%) and by the lack of position papers.

3 3. Discussion Smart Cities, Urban Sensing and Big Data: Mining Geo-location in Social Networks We have identified 3 main domains: (a) data sources, (b) technologies, (c) use scenarios. Let us consider the main contributions to each domain. 4.1 Data Sources The data sources that can be inferred for urban sensing are heterogeneous. Three innovative data sources exist: (a) mobile sensor data about the individual devices, (b) infrastructure sensor data about the context, and (c) social data from social network and other internet services [Zhang et al, 2011]. Data sources can be used independently but the combination of the three kinds can provide a comprehensive understanding of human behavior and its context. Here we focus only on data sources for geo-location from the social network tier and how they have been used in literature. Twitter is a widely-used platform for the real-time social sharing of short textbased messages called tweets. Twitter and smartphone usage reflected same growth, indeed Twitter users interact frequently on mobile devices. As Twitter is easy to use and interactions are short, many users post tweets despite they are engaged in other activities. This gives Twitter data good spatial and temporal coverage because tweets can be automatically geo-located [Mai et Hranac, 2013]. Twitter provides a free real-time streaming API through which a sample of all tweets can be retrieved. The API provides filters that can be set on these data streams to capture tweets within a geographic area or only those containing certain terms. However, data stream is limited to 1% of total tweet volume. So, only a subset of the total tweets can be used. Foursquare is a location-based social network where users can check in to different locations and share them with friends both on Foursquare itself and other social networks. Users can upload pictures at a venue or leave tips on the venue page (e.g. a user may check-in to a hotel and leave a tip about how bad the service is) [Cheng et al, 2011]. Foursquare check-in data is not directly accessible: however, users typically decide to share their check-ins publicly on Twitter, so they can be retrieved via Twitter streaming API. Several other papers use data sets published for research purposes, because no API is available or the social network stopped its service. For example, Gowalla was a location-based social network created in The concept behind the service was to advertise your exact location to all your friends in real-time [Scellato et al, 2010]. Also BrightKite was a social networking website where users could share their location, to post notes and to upload photos. By making check-ins at places, users could see people who is nearby. Now, only already collected data sets are available [Li et Chen, 2009]. Other available services are Momo and Flickr. However, a question could rise: why not to use Facebook, the most popular service? The main reason is that Facebook API can be used to retrieve data from those users who accepted to publish their posts to your application or system, so it is not publicly available.

4 Congresso Nazionale AICA Technologies We discuss here the main technologies used for urban sensing. The SLR shows that related works focus on machine learning techniques. K-means can be used to reveal clusters of common behaviors across land segments. The land use of each cluster can be derived by analyzing the activity vectors of the regions within the cluster. K-means depends on the initial random selected seeds and it needs to specify the number of clusters k (land uses) to identify [Frias-Martinex et al, 2012]. [Lee et al, 2011] used a similar approach, but they also formed a Voronoi diagram using the center points (latitude, longitude) of the K-means results and regarded the formed regions as a set of region of interests, to identify the occurrence of local events. A Self-Organizing Map (SOM) is an unsupervised neural network that reduces the input data dimensionality to be able to represent its distribution as a map. So, SOM forms a map where similar samples are mapped close together. [Frias-Martinex et al, 2012] used SOM to build a map that segments the urban land into geographical areas with different concentrations of tweets in the time period under study. Density-Based Spatial Clustering of Applications with Noise (DBSCAN) has specific characteristics: (a) it is based on the concept of density reachability, producing satisfying results identifying arbitrarily shaped clusters, (b) the number of clusters is not given a priori, and (c) the algorithm tolerates noise, allowing for some data points not to be assigned to any cluster [Villatoro et al, 2013]. The advantage of density based clustering algorithms is that clusters are defined by the density of data points and not by spatial size and form of cluster. Spectral methods for data clustering are popular because of the quality of the clusters that are produced and the simplicity of implementation. Spectral clustering is able to find arbitrarily shaped clusters and does not pose any constraints on them (in contrast to the k-means, for example, which assumes cluster to be convex). It requires anyway the parameter k to define the number of desired clusters [Roslrer et Liebig, 2013]. Mean-shift is a non-parametric clustering technique that detects the modes of an underlying probability distribution from a set of discrete samples. So, mean-shift can be used both as an algorithm to detect local maxima (modes) as well as a clustering technique (areas associated to the modes). [Frias-Martinex et al, 2012] assume that there exists an unobservable underlying probability distribution of where people tweet from. The modes of that distribution are determined to represent urban landmarks or points of interest in the city. 4.3 Use scenarios The identification of use scenarios aims to enable new perspectives in society level, e.g. community healthcare, public safety, city resource management and transportation management.. We here propose a classification for visualization objectives in urban sensing, representing an evolution of existing Business Intelligence (BI) solutions towards Geographic Information Systems (GISs) and Big Data visualization [Stodder, 2013], as shown in Tab. 1.

5 Smart Cities, Urban Sensing and Big Data: Mining Geo-location in Social Networks Class Description BI similarities Objectives Urban characterization The results can be stored for users as a snapshot of a certain point in time. Users examine snapshots to identify changes in data over time, so they must be provisioned and presented consistently so that trends and comparisons are valid It recalls dashboards. The view is static and it is previously defined by data analysts. To visualize and predict social ties and urban structure Spatial discovery Exception alerting It enables users to interact with data through analytical processes. Visual functionality for filtering, comparing, slicing and dicing, drilling down, and correlating data can then be integrated with the users analytical application functions for forecasting, modeling, and statistical, what-if, and predictive analytics. It notifies users of particularly important changes in the data or when situations arise that demand immediate attention. Alerts mean that something important in real-time data or event streams is happening. It recalls On-Line Analytical Processing (OLAP). The view is dynamic and it allows users to navigate data. It recalls event processing in modern Business Activity Monitoring (BAM) solutions, that detect and warn of problems or exceptions in realtime. Tab. 1 Visualization objectives classification To analyze behaviors online, in space and time To detect events or exceptions to standard behaviors, e.g. disasters, diseases, unexpected crowds The three visualization classes reflect papers analyzed by our SLR. Respectively, 30 papers deal with urban characterization, 18 with spatial discovery, 12 with exception alerting. The rest of papers use social networks geo-located data mainly to build recommender systems or user profiling. These papers are single user oriented, so we do not take them into account in following paragraphs because they do not give a broad view of the city Urban characterization [Rosler et Liebig, 2013] provide insights on the activity profiles in urban environments. Clusters identified by Foursquare check-ins help to describe the socio-dynamics of urban districts in different times of the day. [Ferrari et al, 2011] extend the work on activity profiles by providing also mobility patterns that occur in an urban environment and understanding of social commonalities between people. Traditional municipal organizational units such as neighborhoods are studied by Livehoods project from [Cranshaw et al, 2012], who shows that their boundaries do not always reflect the character of life in these areas. [Joseph et al, 2012] mine check-ins to identify groups of people which are of different types (e.g. tourists), communities (e.g. users tightly clustered in space) and interests, and how they use urban space.

6 Congresso Nazionale AICA 2013 By processing Twitter data, [Wakamiya et al, 2011] are able to examine the relation between regions of common crowd activity patterns and major categories of local facilities. [Wakamiya et al, 2012] extend their work and correlate psychological and geospatial proximity of urban areas by borrowing crowd s experiences from geo-tagged tweets, in order to demonstrate that people often rely on geospatial cognition to the real space than the exact physical distance in the real world Spatial discovery [Silva et al, 2012] study social behaviors by monitoring check-ins in Foursquare, [Sagl et al, 2012] by Twitter and Flickr data. They are able to analyze city dynamics spatially and temporally and to identify seasonality in human behaviors. [Cheng et al, 2011] investigate 22 million check-ins across 220,000 users and report a quantitative assessment of human mobility patterns by analyzing the spatial, temporal, social, and textual aspects associated with these footprints Exception alerting [Boettcher et Lee, 2012] introduce EventRadar, a detection system that finds local events such as release parties, musicians in a park, or art exhibitions. [Watanabe et al, 2011] provide a similar system to detect events. [Baldwin et al, 2012] also try to predict events impact. [Mai et Hranac, 2013] assert that tweets can be matched to traffic incidents by examining the content of the tweets for key words and comparing locations of the tweets and incidents. Results are confirmed for areas with sufficient density of Twitter usage. 4. A Big Data-based approach Big data is a popular term used to refer to the exponential growth, availability and use of information. However, as Gartner states ( Big Data doesn t focus only on the high volumes of information, but also on data velocity, that involves streams of data that are produced and processed in real-time, and on data variety, that refers to more types of information to analyze, e.g. social media, context aware data, documents, images, videos, audio, etc. All papers in our review obviously consider data variety because locationbased social networks data is geo-located and unstructured (typically JSON files), however the challenge results from the expansion of all three properties given by Gartner, rather than just data volume or data velocity alone. We here propose a big data-based approach to mining of social network geo-located data. Our approach intends to consider the 3 properties by applying technologies that fit Big Data management. Our literature review found out no papers take into account big data-based system architectures to process variety, velocity and volume of social network data. Here we propose a simple architecture to process this kind of data and possible extensions to support more analytics.

7 Smart Cities, Urban Sensing and Big Data: Mining Geo-location in Social Networks In order to validate our approach, we report a case study about public transport monitoring. First we developed a simulator that transforms timetables from open transport data in Torino (Italy) to a visualization tool that shows planned position of each bus in a specific time (each moving point in Fig. 1). We decided to enrich this visualization and leverage urban sensing. Our specific question was: how to correlate transport planning and urban activity areas?. Exploitation of location-based data from social network is a viable way. To define activity areas in a city we built a tool for spatial discovery that could allow us to model crowds by density clusters within the city and drill down them according to different scales. In order to keep the case study simple, our approach has been to compute density on a fixed grid applied over the map (rectangles with same dimension). However, what kind of data could be used? We exploited Twitter streaming API, collected geo-located data in Torino area in real-time and clustered them on the map for each specific time range (1 hour). Fig. 1 Public transport simulator To retrieve Twitter data we implemented FluenTD agents by Node.js on a virtual machine in Amazon Elastic BeanStalk. FluenTD ( is an open-source log collector that enables to have a logging architecture with more than an hundred types of systems, by treating logs as JSON. Node.js ( is a platform built on JavaScript for easily building fast, scalable network applications. It uses an event-driven, non-blocking I/O model that makes it lightweight, efficient, and oriented to data-intensive real-time applications that run across distributed devices. FluenTD and Node.js integration allows to build real-time collection and filtering of geo-located tweets, and their storage in TreasureData ( TreasureData is a Big Data as a Service cloud solution that offers a time series, columnar, Hadoop-based, schema-free data warehouse stored on Amazon S3. It allows you to access data using Hive query language ( by JDBC. An Extraction-Transformation-Loading (ETL) process access data in TreasureData, processes it and uploads it in CartoDB every 5 minutes. CartoDB is a database as a service cloud solution, based on a PostgreSQL database with GIS extension. As PostgreSQL databases can use

8 Congresso Nazionale AICA 2013 extensions to run data mining algorithms, we decided to apply density-based clustering online, whenever the user changes the zoom scale in the map. Fig. 2 shows the final architecture for our prototype. As this solution intends to implement Service Oriented Architecture (SOA), it has two main advantages: (a) it can be easily extended, and (b) a component can be replaced by others because of decoupling. For example, the data stored in the data mart can be used to perform different analytics or to build an alerting system. As SAP Hana has a Predictive Analysis Library that offers native support to DBSCAN, we could use SAP Hana to implement it as a supplementary layer between TreasureData and CartoDB. If we want to easily implement Latent Dirichelet Allocation we could replace TreasureData with Mahout on a Hadoop cluster. The ETL process, that may represent a bottleneck in terms of performances, can be replaced by ad hoc solutions, e.g. interfaces developed in Node.js to provide data filtering and storage in the data mart. Fig. 2 A Service Oriented Architecture for Big Data management Fig. 4 shows early results of our prototype. It considers tweets collected from 8 AM to 9 AM on July 10 th, Color density of rectangles change according to tweet density in the same area. Fig. 4 - Tweet density in Torino Final integration between the crowd modeling tool and the public transport simulator allows spatial discovery of urban areas not covered by transport service during peaks of presence of people who may need to move from one place to another within the city.

9 5.Conclusion Smart Cities, Urban Sensing and Big Data: Mining Geo-location in Social Networks Our SLR demonstrated the exponential growth in number of papers about this trending topic. The high number of case studies and instrument development publications reflects the experimental approach to the issue and it may also lead developers and freelancers to provide smart solutions to contextual problems in the city. Most of related works focus on validation of data mining techniques, rather than their application in real use scenarios. Our prototype demonstrated that the integration of urban sensing and open data can help municipalities to reveal enhanced insights about their services to citizens. So, future research should move from validation of techniques to their real application in Smart Cities by exploiting Big Data technologies, thus providing real time analytics to municipalities and end users. Our future works intend to integrate different data sources, such as sensors, and more social media, in order to provide deeper insights for urban sensing. Furthermore, our system needs to scale up and validate performances against user needs. Next step will be the development of a service orchestration layer to provide complete validation of our Service Oriented Architecture. References [Baldwin et al, 2012] Baldwin, T., Cook, P., Han, B., Harwood, A., Karunasekera, S., & Moshtaghi, M. (2012, April). A support platform for event detection using social intelligence. 13th Conference of the European Chapter of the Association for Computational Linguistics (pp ). [Boettcher et Lee, 2012] Boettcher, A., & Lee, D. (2012, November). EventRadar: A Real-Time Local Event Detection Scheme Using Twitter Stream. In Green Computing and Communications (GreenCom), IEEE International Conference on (pp ). [Chen et al, 2013] Chen, T., Kaafar, M. A., & Boreli, R. (2013). The Where and When of Finding New Friends: Analysis of a Location-based Social Discovery Network. International Conference On Weblogs And Social Media [Cheng et al, 2011] Cheng, Z., Caverlee, J., Lee, K., & Sui, D. Z. (2011). Exploring Millions of Footprints in Location Sharing Services. ICWSM, 2011, [Cranshaw et al, 2012] Cranshaw, J., Schwartz, R., Hong, J. I., & Sadeh, N. M. (2012, June). The Livehoods Project: Utilizing Social Media to Understand the Dynamics of a City. In ICWSM. [Egger et al, 2008] Egger, M., Smith, G. D., & Altman, D. (Eds.). (2008). Systematic reviews in health care: meta-analysis in context. Wiley. [Ferrari et al, 2011] Ferrari, L., Rosi, A., Mamei, M., & Zambonelli, F. (2011, November). Extracting urban patterns from location-based social networks. 3rd ACM SIGSPATIAL International Workshop on Location-Based Social Networks (pp. 9-16) [Frias-Martinez et al, 2012] Frias-Martinez, V., Soto, V., Hohwald, H., & Frias- Martinez, E. (2012, September). Characterizing Urban Landscapes using Geolocated Tweets IEEE International Conference on Social Computing (pp ).

10 Congresso Nazionale AICA 2013 [Joseph et al, 2012] Joseph, K., Tan, C. H., & Carley, K. M. (2012, September). Beyond local, categories and friends: clustering foursquare users with latent topics. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (pp ) [Lee, 2012] Lee, C. H. (2012). Mining spatio-temporal information on microblogging streams using a density-based online clustering method. Expert Systems with Applications, 39(10), [Lee et al, 2011] Lee, R., Wakamiya, S., & Sumiya, K. (2011). Discovery of unusual regional social activities using geo-tagged microblogs. World Wide Web, 14(4), [Li et Chen, 2009] Li, N., & Chen, G. (2009, August). Analysis of a location-based social network. IEEE International Conference on CSE'09 (Vol. 4, pp ).. [Mai et Hranac, 2013] Mai, E., & Hranac, R. (2013, January). Twitter Interactions as a Data Source for Transportation Incidents. In Transportation Research Board 92nd Annual Meeting (No ). [Rosler et Liebig, 2013] Rösler, R., & Liebig, T. (2013). Using Data from Location Based Social Networks for Urban Activity Clustering. In Geographic Information Science at the Heart of Europe (pp ). Springer International Publishing. [Sagl et al, 2012] Sagl, G., Resch, B., Hawelka, B., & Beinat, E. (2012). From Social Sensor Data to Collective Human Behaviour Patterns: Analysing Spatio-Temporal Dynamics in Urban Environments. GI-Forum: Geovisualization, Society and Learning. [Scellato et al, 2010] Scellato, S., Mascolo, C., Musolesi, M., & Latora, V. (2010, June). Distance matters: geo-social metrics for online social networks. In Proceedings of the 3rd conference on Online social networks (pp. 8-8). USENIX Association. [Silva et al, 2012] Silva, T. H., Melo, P. O., Almeida, J. M., Salles, J., & Loureiro, A. A. (2012, November). Visualizing the invisible image of cities. In Green Computing and Communications (GreenCom), 2012 IEEE International Conference on (pp ) [Stodder, 2013] Stodder, D. (2013). Data Visualization and Discovery for Better Business Decisions. TDWI Best Practices Report. TDWI Research [Villatoro et al, 2013] Villatoro, D., Serna, J., Rodríguez, V., & Torrent-Moreno, M. (2013). The TweetBeat of the City: Microblogging for Discovering Behavioural Patterns during the MWC2012. Citizen in Sensor Networks (pp ). Springer Berlin [Wakamiya et al, 2011] Wakamiya, S., Lee, R., & Sumiya, K. (2011, November). Crowd-based urban characterization: extracting crowd behavioral patterns in urban areas from twitter. ACM SIGSPATIAL Workshop on LBSNs (pp ) [Wakamiya et al, 2012] Wakamiya, S., Lee, R., & Sumiya, K. Measuring Crowdsourced Cognitive Distance between Urban Clusters with Twitter for Socio-cognitive Map Generation. International Conference On Emerging Databases [Watanabe et al, 2011] Watanabe, K., Ochi, M., Okabe, M., & Onai, R. (2011, October). Jasmine: a real-time local-event detection system based on geolocation information of microblogs. ACM conference on Information and knowledge management [Zhang et al, 2011] Zhang, D., Guo, B., & Yu, Z. (2011). The emergence of social and community intelligence. Computer, 44(7),