Urban sensing using mobile phone network data. Ubicomp 2011 Tutorial

Transcription

1 Urban sensing using mobile phone network data Advisory Research Staff Member Smarter Cities Technology Centre IBM Research, Dublin, Ireland Ubicomp 2011 Tutorial 2010 IBM Corporation

2 Outline of the Tutorial 8:30 Introduction to urban dynamics Mobile phone network data Comparative datasets 10:00 Coffee break 10:30 Filtering techniques Processing techniques Open challenges 12:00 Lunch

3 Context 50% of the globe s population live in urban areas (just 0.4% of the Earth s surface) 70% are projected to do so by 2050 The greatest wave of urbanization is yet to come Great opportunity for improving people life stiles A potential economic, health and environmental disaster

4 Pervasive technology datasets as digital footprints Pervasive technologies datasets are a way to understand how people use the city's infrastructure Mobility (transportation mode) Consumption (energy, water, waste) Environmental impact (noise, pollution) Human behavior Pervasive technologies datasets Built environment Natural environment Services Planning and Management

5 Urban dynamics Understanding the urban dynamics allows Improving city s services Optimizing planning Reducing operational costs Human behavior Pervasive technologies datasets Creating feedback loops with citizens to reduce energy consumption and environmental impact Built environment Natural environment Services Planning and Management

6 Potentials Possibility to study micro and macro behaviors Mobile phone network data Data is becoming more and more available (mobile technologies increasingly adopted by the population) Privacy concerns in the use of personal data only partially addressed by the EC (*) (*) Direc*ve 2002/58/EC of the European Parliament and of the Council of 12 July 2002 concerning the processing of personal data and the protec*on of privacy in the electronic communica*on sector (Direc*ve on privacy and electronic communica*ons) Technology Review, Mobile Data: A Gold Mine for Telcos, The Wall Street Journal, The really smart phone, SB html

7 Urban sensing using mobile phone network data Mobile phone network data is being used to Estimate population distribution Estimate types of activities in different parts in the city Estimate how areas in the cities are connected Estimate residential and working areas Estimate commuting patterns

8 References Urban Sensing using mobile phone network data Workshops on the theme Analysis of Mobile Phone Datasets and networks (NetMob), Pervasive Urban Applications, Mobility F. Calabrese, M. Colonna, P. Lovisolo, D. Parata, C. Ratti, Real-Time Urban Monitoring Using Cell Phones: a Case Study in Rome, IEEE Transactions on Intelligent Transportation Systems, 2011 M.C. González, C.A. Hidalgo and A.-L. Barabási, Understanding individual human mobility patterns, Nature, 453, , V.A. Traag, A. Browet, F. Calabrese, F. Morlot, Social Event Detection in Massive Mobile Phone Data Using Probabilistic Location Inference, International Conference on Social Computing, 2011 S. Isaacman, R. Becker, R. Cáceres, S. Kobourov, M. Martonosi, J. Rowland, and A. Varshavsky, Identifying Important Places in People's Lives from Cellular Network Data, International Conference on Pervasive Computing, 2011 F. Calabrese, G. Di Lorenzo, L. Liu, C. Ratti, Estimating Origin-Destination flows using opportunistically collected mobile phone location data from one million users in Boston Metropolitan Area, IEEE Pervasive Computing, 2011 F. Calabrese, F. Pereira, G. Di Lorenzo, L. Liu, and C. Ratti, The geography of taste: analyzing cell-phone mobility and social events, International Conference on Pervasive Computing, 2010 Urban activity V. Soto, E. Frias-Martinez, Robust Land Use Characterization of Urban Landscapes using Cell Phone Data, First Workshop on Pervasive Urban Applications, 2011 J. Reades, F. Calabrese, A. Sevtsuk, and C. Ratti, C., Cellular Census: Explorations in Urban Data Collection, IEEE Pervasive Computing, 6(3), 30-38, 2007 F. Girardin, A. Vaccari, A. Gerber, A. Biderman, C. Ratti, Quantifying urban attractiveness from the distribution and density of digital footprints, International Journal of Spatial Data Infrastructures, 4, , 2009 Social networks R. Lambiotte, V.D. Blondel, C. de Kerchove, E. Huens, C. Prieur, Z. Smoreda, P. Van Dooren, Geographical dispersal of mobile communication networks, Physica A: Statistical Mechanics and its Applications, 387, , 2008 V.D. Blondel, G. M. Krings, I. Thomas, Regions and borders of mobile telephony in Belgium and around Brussels, Brussels Studies 42, 2010 F Calabrese, D Dahlem, A Gerber, D Paul, X Chen, J Rowland, C Rath, C Ratti, The Connected States of America: Quantifying Social Radii of Influence, International Conference on Social Computing, 2011 F. Calabrese, Z. Smoreda, V. Blondel, C. Ratti, The Interplay Between Telecommunications and Face-to-Face Interactions-An Initial Study Using Mobile Phone Data, PLoS ONE, 2011

9 Outline of the Tutorial Introduction to urban dynamics Mobile phone network data Event and Network driven Location and Communication Spatial and Temporal resolution Comparative datasets Census and surveys Land use and Points of Interest Filtering techniques Normalization Stop detection Processing techniques Aggregate Land use inference Regional partitioning Individual Home and work location detection Origin/Destination Attendance to social events Face-to-face meetings Open challenges

10 Mobile phone network data Data generated by the interaction between mobile phone and the serving telecommunication network Triggered by Events (call/sms/internet usage) Network Periodic Mobility-driven Different device location information is available at different levels (interfaces) of the telecommunication network Cell tower Cell sector Triangulated position Data can be aggregated at different spatial and temporal levels Urban Sensing using mobile phone network data

11 Event-driven mobile phone network data Communication (Call Detail Record) Originating User id Terminating User id Type of communication (Call, SMS) Time of event Length (for calls) Internet usage (IP Detail Record) User id Type of website Time of event Number of bytes transmitted It can be associated to the cellphone tower(s) used during the interaction

12 Network-driven mobile phone network data Periodic location update Generated on a periodic base and providing information on which cell tower the phone is connected two (even if not in call, just turned on) Mobility location update Generated when the phone moves between two Location Areas Handover Generated when a phone involved in a call moves between two Cell areas Data User id Time of event Cellphone tower(s) used during the interaction LA1 LA3 LA2 LA4

13 Location information Location information can be extracted as part of the interaction data In most cases (e.g. Call Detail Records) Cell tower Cell sector Accuracy around 500m in Urban areas Cell tower Cell sector

14 Location information For data collected at lower levels in the network Timing Advance (TA) Received Signal Strength (RSS) from all surrounding cell towers Using propagation models and irradiation diagrams, the mobile phone position can be estimated by finding the point that minimizes the mean square error between measured and estimated mean power received by all base stations [1]. Accuracy around 150m in Urban areas Timing Advance (TA) Received Signal Strength (RSS) [1] F. Calabrese, M. Colonna, P. Lovisolo, D. Parata, C. Ratti, Real-Time Urban Monitoring Using Cell Phones: a Case Study in Rome, IEEE Transactions on Intelligent Transportation Systems, 2011.

15 Considerations Data can be aggregated at different spatial and temporal scales E.g. Cell tower aggregation Number of calls Erlang (total communication time) Number of sms Number of Handovers Number of Location updates Availability Individual data is rarely available in real time Exception if additional hardware is installed Usually available the day after Aggregated data can be more easily accessible Potentially in real time or with low delay Volume Aggregated data can be easily manageable Individual data might be difficult to manage Solution: taking subset of users Problem: how to select a good sample?

16 Comparison Advantages Disadvantages Applications Cell tower statistics Easy to manage, Possibly in real time No information on users mobility Land use estimation, Population density estimation Aggregated CDR Easy to manage No individual interaction information Connection between places, Regional partitioning Individual CDR Individual communication patterns Large dataset, Mostly not real time Social network analysis Individual CDR with Cell tower location info Individual communication and mobility patterns Large dataset, Mostly not real time Mobility analysis between large areas Individual Event-driven triangulated location Individual mobility patterns, Possibly in real time Large dataset, Possibly need for special hardware in telco Origin destination, transportation mode Individual Networkdriven data Individual mobility patterns, Possibly in real time Large datasets, Possibly need for special hardware in telco Useful for mobility analysis between large areas

17 Outline of the Tutorial Introduction to urban dynamics Mobile phone network data Event and Network driven Location and Communication Spatial and Temporal resolution Comparative datasets Census and surveys Land use and Points of Interest Filtering techniques Normalization Stop detection Processing techniques Aggregate Land use inference Regional partitioning Individual Home and work location detection Origin/Destination Attendance to social events Face-to-face meetings Open challenges

18 Comparative datasets A large percentage of a population can be analyzed by looking at mobile phone network data Mobile phone penetration is high A telecom operator can have good share of the market Comparative datasets are useful to Validate findings extracted from analysis of mobile phone data Define scaling factors to extend results to overall population Augment information about urban space, useful to extract higher level patterns Mainly used datasets in urban sensing Census Transport surveys Land use Points of Interest

19 Census and survey data Interesting datasets Population distribution Age Income Home-to-work flow Advantages Spatial resolution (census block) Disadvantages Updated usually every 5/10 years, so very soon outdated Only some questions are asked

20 Land use and Points of Interest Land use Characterizes an area based on how it has been planned to be used Different land use categories defined Urban Sensing using mobile phone network data Points of Interest Directory of businesses by category and location Many different sources: Businesses registry Yellow pages Yelp which might provide different information In most comparisons, categories are aggregated in super-categories

21 Challenges and limitations in comparing datasets Different collection period Different spatial units Example Census data aggregated at block, track, county level Mobile phone network data aggregated at cell tower level Solutions Aggregating at largest spatial unit Spatial interpolation

22 Outline of the Tutorial Introduction to urban dynamics Mobile phone network data Event and Network driven Location and Communication Spatial and Temporal resolution Comparative datasets Census and surveys Land use and Points of Interest Filtering techniques Normalization Stop detection Processing techniques Aggregate Land use inference Regional partitioning Individual Home and work location detection Origin/Destination Attendance to social events Face-to-face meetings Open challenges

23 Filtering techniques spatial uncertainty Area covered by a cell tower/sector We assign a user to the centroid of the cell area Voronoi tessellation, based on location of cell tower [1] Best serving cell map, based on simulated coverage [2] Taking into account cell sector Taking into account propagation Voronoi tessellation Best serving cell map [1] M.C. González, C.A. Hidalgo and A.-L. Barabási, Understanding individual human mobility patterns, Nature 453, , 2008 [2] F. Girardin, A. Vaccari, A. Gerber, A. Biderman, C. Ratti, Quantifying urban attractiveness from the distribution and density of digital footprints, International Journal of Spatial Data Infrastructures, 4, , 2009

24 Filtering techniques spatial uncertainty Area covered by a cell tower/sector We assign a user a probability to be in a given location [1] A propagation model is used to assign a probability a user staying in a location to be connected to a particular cell tower Allows for multiple cell towers to be covering a same location Propagation model [1] V.A. Traag, A. Browet, F. Calabrese, F. Morlot, Social Event Detection in Massive Mobile Phone Data Using Probabilistic Location Inference, International Conference on Social Computing, 2011.

25 Filtering techniques determining important places Leveraging consecutive location data to improve accuracy Consecutive events associated to different close by locations Averaging location among consecutive measurements within a given spatial and temporal window [1] [1] F. Calabrese, F. Pereira, G. Di Lorenzo, L. Liu, C. Ratti, The geography of taste: analyzing cell-phone mobility and social events, In International Conference on Pervasive Computing, 2010.

26 Filtering techniques determining important places Leveraging historical location data to improve accuracy Clustering visited cell towers Hartigan s leader algorithm [1] [1] S. Isaacman, R. Becker, R. Cáceres, S. Kobourov, M. Martonosi, J. Rowland, and A. Varshavsky, Identifying Important Places in People's Lives from Cellular Network Data, International Conference on Pervasive Computing (Pervasive), 2011.

27 Outline of the Tutorial Introduction to urban dynamics Mobile phone network data Event and Network driven Location and Communication Spatial and Temporal resolution Comparative datasets Census and surveys Land use and Points of Interest Filtering techniques Normalization Stop detection Processing techniques Aggregate Land use inference Regional partitioning Individual Home and work location detection Origin/Destination Attendance to social events Face-to-face meetings Open challenges

28 Land use inference Urban Sensing using mobile phone network data Goal: Understand activities in the city from telecommunication usage patterns Augment existing built environment data collection & analysis methods (census, business registrations, etc.) at low cost and with very low latencies [1] J. Reades, F. Calabrese, A. Sevtsuk, C. Ratti, Cellular Census: Explorations in Urban Data Collection, IEEE Pervasive Computing, 2007 [2] V. Soto, E. Frias-Martinez, Robust Land Use Characterization of Urban Landscapes using Cell Phone Data, First Workshop on Pervasive Urban Applications, 2011

29 Categories of activities Classical time series analysis can be performed E.g. principal component analysis Clustering of time series can classify places based on usage Time (h)

30 Regional partitioning Goal: Partition space based on level of human interactions Modeling the effect of geography on human mobility and interactions Mobile phone users location at call time can be used to infer origin and destination of calls Human interactions decrease as distance increases, following a gravity-like behavior [1] Exception emerges due to Geographical features (e.g. rivers) [2] Administrative borders Cultural differences [1] R. Lambiotte, V.D. Blondel, C. de Kerchove, E. Huens, C. Prieur, Z. Smoreda, P. Van Dooren, Geographical dispersal of mobile communication networks, Physica A: Statistical Mechanics and its Applications, 387, , [2] C Ratti, S Sobolevsky, F Calabrese, C Andris, J Reades, M Martino, R Claxton, S H Strogatz, Redrawing the map of Great Britain from a network of human interactions, PLoS ONE, 2010.

31 City scale partitioning We can aggregate interaction events to create a network of places nodes are locations (e.g. cell towers) edges between nodes exists if interactions happen between people connected to two cell towers The weighted graph can be partitioned in communities using standard network analysis tools Modularity optimization We can detect which areas in the city are most connected where interaction borders exist How borders change over time [1] [1] F. Walsh, A. Pozdnoukhov, Spatial structure and dynamics of urban communities, First Workshop on Pervasive Urban Applications, 2011.

32 Urban Sensing using mobile phone network data Country scale partitioning Findings Spatial cohesiveness of regions State boundaries emerge in most of the cases [1] Metropolitan areas (e.g. NYC, LA) define new regions Some areas merge as level of interaction is higher than expected [1] F Calabrese, D Dahlem, A Gerber, D Paul, X Chen, J Rowland, C Rath, C Ratti, The Connected States of America: Quantifying Social Radii of Influence, International Conference on Social Computing (Socialcom), [2] V.D. Blondel, G. M. Krings, I. Thomas, Regions and borders of mobile telephony in Belgium and around Brussels, Brussels Studies 42, 2010

33 Outline of the Tutorial Introduction to urban dynamics Mobile phone network data Event and Network driven Location and Communication Spatial and Temporal resolution Comparative datasets Census and surveys Land use and Points of Interest Filtering techniques Normalization Stop detection Processing techniques Aggregate Land use inference Regional partitioning Individual Home and work location detection Origin/Destination Attendance to social events Face-to-face meetings Open challenges

34 Home and work location estimation Urban Sensing using mobile phone network data Goal: determine users home and work location Leveraging location data over several days to increase precision in estimation Presence criteria Number of times a cell tower was contacted Length (time) of stay in a location Home location Most presence during evenings Work location Most presence during weekday morning/afternoon Excluding home location With low number of evening events [1] F. Calabrese, G. Di Lorenzo, L. Liu, C. Ratti, Estimating Origin-Destination flows using opportunistically collected mobile phone location data from one million users in Boston Metropolitan Area, IEEE Pervasive Computing, [2] S. Isaacman, R. Becker, R. Cáceres, S. Kobourov, M. Martonosi, J. Rowland, and A. Varshavsky, Identifying Important Places in People's Lives from Cellular Network Data, International Conference on Pervasive Computing (Pervasive), 2011.

35 Home location validation Validation data from US Census population estimates at census track level 0 estimated tract population density (log10) census tract population density (log10) Validation data from 19 volunteers errors in miles

36 Estimating Daily mobility Inferring daily trips Distance between any two different visited locations [1] Daily range of mobility, related to where people live [2] [1] M.C. González, C.A. Hidalgo and A.-L. Barabási, Understanding individual human mobility patterns, Nature 453, , 2008 [2] S. Isaacman, R. Becker, R. Cáceres, S. Kobourov, M. Martonosi, J. Rowland, and A. Varshavsky, Identifying Important Places in People's Lives from Cellular Network Data, International Conference on Pervasive Computing (Pervasive), 2011.

37 Urban Sensing using mobile phone network data Origin and destination of trips We can map origin and destination of detected trips and so: Count number of trips for any time of day [1] Analyze attractiveness of an area Number of different places people come from x 10 2 number of trips Sun Mon Tue Wed Thu Fri Sat 1.5 Sun Mon Columbus Day Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat 1.7 [1] F. Calabrese, G. Di Lorenzo, L. Liu, C. Ratti, Estimating Origin-Destination flows using opportunistically collected mobile phone location data from one million users in Boston Metropolitan Area, IEEE Pervasive Computing, 2011.

38 Urban Sensing using mobile phone network data How social events impact mobility in the city Goal: Modeling and predicting non-routine additive origin-destination flows in the city Estimated home location Event duration User stop Overlap time > 70% Time Attendance Inference [1] F. Calabrese, F. Pereira, G. Di Lorenzo, L. Liu, C. Ratti, The geography of taste: analyzing cell-phone mobility and social events, In International Conference on Pervasive Computing, [2] V.A. Traag, A. Browet, F. Calabrese, F. Morlot, Social Event Detection in Massive Mobile Phone Data Using Probabilistic Location Inference, International Conference on Social Computing, 2011.

39 Methodology for detecting and predicting travel demand

40 Integrating social and mobility information Goal: Inferring face-to-face meetings Keeping social interaction induce the need for social travels and face-to-face meetings Integrating calling and location pattern help inferring such events!"#$%&!"#$%'!!!"#$%'(&!! )*+,$%- )*+,$%.!"#$%&"'()*"+,%-*,)(+'$ [1] F. Calabrese, Z. Smoreda, V. Bondel, C. Ratti, Interplay between telecommunications and face-to-face interactions an study using mobile phone data, PLoS ONE, 2011.

41 We discovered that people calling while connected to the same cell tower (co-location) are a good proxy for face-to-face meetings Validation: People tend to interact much more just before and after this event Number of inferred face-to-face meetings decreases with the users home distance #colocations #calls total call time distribution of calls number (log10) normalized time between consecutive co locations distance between home locations (km) We can predict when and where people will be meeting from the call interactions

42 Outline of the Tutorial Introduction to urban dynamics Mobile phone network data Event and Network driven Location and Communication Spatial and Temporal resolution Comparative datasets Census and surveys Land use and Points of Interest Filtering techniques Normalization Stop detection Processing techniques Aggregate Land use inference Regional partitioning Individual Home and work location detection Origin/Destination Attendance to social events Face-to-face meetings Open challenges

43 Open challenges Limitations of event-driven data Limitations in spatial accuracy Managing uncertainties Find comparative datasets Dealing with privacy and anonymity Mobility/communication interplay Real Time data acquisition and processing

44 Open challenges Limitation of event-driven data It might be important to have very frequent location data for certain types of applications Proposed approaches so far Sampling only highly active users Problem: be sure these users represent a good sample High communication has been found to be correlated to high mobility [1] Sampling Smartphone Internet usage data Generally lower inter-event time [2] Problem: smartphone users bahavior could not represent a general sample Network-driven data Periodic sampling, independent on events Not too good for short term mobility Mobility-based sampling Good for analyzing mobility between large areas (Location Areas can be quite large) [1] T. Couronne, Z. Smoreda, A.-M. Olteanu, Chatty Mobiles: Individual mobility and communication patterns, NetMob, 2011 [2] F. Calabrese, F. Pereira, G. Di Lorenzo, L. Liu, C. Ratti, The geography of taste: analyzing cell-phone mobility and social events, In International Conference on Pervasive Computing, 2010.

45 Open challenges Limitation in spatial accuracy It might be important to have very precise location data for certain types of applications Examples Determining accurate location Determining route Determining transporation mode Proposed solutions Look at history for recurring locations Look at handover during call [1] [1] R.A. Becker, R.Caceres, K.Hanson, J.M. Loh, S.Urbanek, A. Varshavsky C. Volinsky, Route Classification Using Cellular Handoff Patterns, Ubicomp, 2011.

46 Open challenges Managing uncertainties As uncertainties in the user s status in time and space can be relatively large, it is important to provide reliable and uncertain-aware results Proposed solutions Estimating undertainties in users position [1] [1] T. Couronné, A-M Olteanu, Z. Smoreda, Urban mobility: velocity and uncertainty in mobile phone data, Workshop on Social Connections in the Urban Space, 2011

47 Open challenges Find comparative datasets Traditional city data (e.g. census) are collected using different methods sampling time collection years Proposed alternatives Self reported data Flickr [1] Social networking data FourSquare [2] [1] F. Girardin, F. Calabrese, F. Dal Fiore, C. Ratti, Digital Footprinting: Uncovering Tourists with User-Generated Content, IEEE Pervasive Computing, [2] A. Bawa-Cavia, Sensing The Urban: Using location-based social network data in urban analysis, PURBA, 2011.

48 Open challenges Dealing with privacy and anonimity Using individual data, even if anonymized, it is possible to detect important information from users E.g. Home and work location More than one top location -> anonymity not preserved [1] Proposed solutions Location obfuscation [2] K-anonymity for trajectories [1] H. Zang, J. Bolot, Anonymization of location data does not work: A large-scale measurement study, ACM Mobicom, 2011 [2] D.Quercia, I. Leontiadis, L. McNamara, C. Mascolo, J. Crowcroft, SpotME If You Can: Randomized Responses for Location Obfuscation on Mobile Phones, ICDCS, [3] A. Monreale, R. Trasarti, D. Pedreschi, C. Renso, V. Bogorny: C-safety: a framework for the anonymization of semantic trajectories. Transactions on Data Privacy, 2011

49 Open challenges Mobility/communication interplay Leveraging communication information, mobility understanding and prediction could be improved Proposed solutions Face to face meeting prediction [1] Link prediction [2] [1] F. Calabrese, Z. Smoreda, V. Bondel, C. Ratti, Interplay between telecommunications and face-to-face interactions an study using mobile phone data, PLoS ONE, [2] D. Wang, D. Pedreschi, C. Song, F. Giannotti, A-L Barabási, Human Mobility, Social Ties, and Link Prediction, KDD 2011

50 Open challenges Real time data acquisition and processing Many urban sensing applications are useful if results are presented in real time or near-real time Traffic monitoring Event management Problem Data is acquired and pushed to databases, usually not in real time Massive data requires ad-hoc algorithms and platforms to be processed in real time Proposed solutions Streaming platforms [1,2] [1] C. Kaiser, A. Pozdnoukhov, Modelling City Population Dynamics from Cell Phone Usage Data Streams, NetMob, 2011 [2] L. Gasparini, E. Bouillet, F.Calabrese, O.Verscheure, System and Analytics for Continuously Assessing Transport Systems from Sparse and Noisy Observations: Case Study in Dublin, IEEE International Conference on Intelligent Transportation Systems, 2011

51 Summary of the tutorial In order to make city s services more efficient we need to understand how people use the city infrastructure Pervasive technologies datasets allow to infer micro and macro behaviors of a population Mobile phone network data represents a very valuable source of human behaviour information Research needed in Inferring behavioral patterns Building analytics and systems to process massive datasets and automaticallty extract patterns Building control systems able to make use of inferred patterns to optimize city services