Big data for beginners An introduction for developmental professionals and researchers

Transcription

1 Big data for beginners An introduction for developmental professionals and researchers Sriganesh Lokanathan, LIRNEasia Research relevant to broadband policy and regulatory processes New Delhi, India 21 August 2014 This work was carried out with the aid of a grant from the International Development Research Centre, Canada and the Department for International Development UK..

2 Learning outcomes for today Some understanding of what big data means An appreciation for some of the developmental opportunities from leveraging big data using different sources A brief overview of the spillover effects of the big data phenomenon An appreciation of some of the analytical and other challenges concerning the use of big data for development 2

3 What is big data? 3

4 Big Data The Vs VELOCITY Rate of change of data VOLUME Size of data VARIETY Different forms of data VERACITY Uncertainty of data VALUE Potential value of data 4

5 What has facilitated the rise of big data? Vast drops in the cost of storing and retrieving information Exponential growth in computer power and memory data can reside in persistent memory instead of disk and tape Major improvements in techniques for performing machine learning and reasoning 5

6 Sources of big data Administrative data E.g. digitized medical records, insurance records, tax records, etc. Commercial transactions E.g. Bank transactions, credit card purchases, supermarket purchases, online purchases, etc. Sensors and tracking devices E.g. road and traffic sensors, climate sensors, equipment & infrastructure sensors, mobile phones, satellite/ GPS devices, etc. Online activities/ social media E.g. online search activity, online page views, blogs/ FB/ twitter posts, online audio/ video/ images, etc. 6

7 Big data isn t necessarily a new area: the census is the original big data First recorded census occurred in Egypt (circa 3340 BC) But the tools didn t exist to fully exploit them The first tools invented for the 1890 US census It took 7 years to process the 1880 census data Herman Hollerith invents the tabulating machine to process the 1890 census data Source: Adam Schuster 7

8 If we want comprehensive coverage of the population, what are the sources of big data in developing economies? Administrative data? E.g. digitized medical records, insurance records, tax records, etc. Commercial transactions? E.g. Bank transactions, credit card purchases, supermarket purchases, online purchases, etc. Sensors and tracking devices? E.g. road and traffic sensors, climate sensors, equipment & infrastructure sensors, mobile phones, satellite/ GPS devices, etc. Online activities/ social media? E.g. online search activity, online page views, blogs/ FB/ twitter posts, online audio/ video/ images, etc. 8

9 Currently only mobile network big data has the widest possible population coverage 9

10 LIRNEasia s Big Data for Development (BD4D) Research LIRNEasiahas negotiated access to historical and anonymized telecom network big data from multiple operators in Sri Lanka In the current research cycle we are conducting exploratory research on answering a few social science questions related to mobility and connectedness developing a framework with privacy and self-regulatory guidelines for the collection, use and sharing of mobile phone data. 10

11 The data sets Multiple mobile operators in Sri Lanka have provided LIRNEasia access to 4 different types of meta-data: Call Detail Records (CDRs) Records of calls, SMS-es, Internet access Airtime recharge records Data sets do not include any Personally Identifiable Information (PII). All phone numbers are anonymized and LIRNEasiadoes not maintain any mappings of identifiers to original phone numbers 11

12 What are some types of big data captured by mobile network operators? Call Detail Record (CDR) Records of all calls made and received by a person created mainly for the purposes of billing Similar records exist for all SMS-essent and received as well as for all Internet sessions The Cell ID in turn has a lat-long position associated with it. 12

13 What are some types of big data captured by mobile network operators (contd.)? Airtime reload records Records of all airtime reloads performed by prepaid SIMs Each row corresponds to a record of one person s activity: 13

14 How can we use mobile network big data for development? 14

15 We can understand the mobility patterns of the population Low Volume of people High 15

16 We can understand land use characteristics People leave digital traces when they use communication devices. Diurnal patterns of user activity in different locations can be used to classify land use characteristics Euclidean distance between two time series is used to cluster base stations in an unsupervised manner using k- means algorithm 16

17 With even a simplistic clustering into just two regions we can clearly see the commercial areas Cluster 1 corresponds to more commercial area Cluster 2 corresponds to more residential (and mixed-used regions) 17

18 We can understand the geospatial distribution of the social networks in the country Each link represents the normalized volume of calls between two DSDs Divisional Secretariat Division (DSD) is a third level administrative division; 331 in total in LK Low No. of calls High 18

19 We can understand community structures The social network is segregated such that overlapping connections between communities are minimized using a modularity approach For Sri Lanka, the optimal number of communities discovered by the algorithm was 11 19

20 How much do these communities mesh with existing administrative boundaries? Southern and Northern provinces have the highest similarity to their respective provincial boundaries. 14

21 How much do these communities mesh with existing administrative boundaries (contd.)? Colombo district is clustered as a single community and Gampaha is merged with North Western Province 15

22 We can leverage the differential patterns of airtime recharge. Average recharge amount Higher household income Lower household income Recharge frequency 22

23 to create poverty maps An example from Cote d Ivoir Source: ThoralfGutierrez, Gautier Krings, Vincent D. Blondel, (2013). Evaluating socio-economic state of a country analyzing airtime credit and mobile phone datasets. Available at Average airtime recharge 23

24 What s the use of such work? Can bring timely evidence into the policy making process in developing economies 24

25 An example of timely policy-relevant evidence The image on the left depicts relative density of people in Colombo city and the surrounding regions at 1300 compared to 0000 (midnight the previous day) on a normal weekday. The yellow to red colors depict areas whose density has increased relative to midnight. The blue color depicts areas whose density has decreased relative to midnight (the darker the blue, the greater the loss in density). The clear areas are those where the overall density has not changed. 25

26 Our findings closely match results from expensive & infrequent transportation surveys 26

27 Taking a broader view: the overall process of leveraging mobile network big data for development Mobile network big data (CDRs, Internet access usage, airtime recharge records) Construct behavioral variables (i) Mobility variables (ii) Social variables (iii) Consumption variables Other data sources (i) Census data (ii) HIES data (iii) Survey maps (iv) Transportation schedules (v) ++++ Analytics Insights (i) Urban planning (ii) Crisis management & DRR (iii) Health monitoring & planning (iv) Socio-economic monitoring & planning 27

28 What are other potential data sources for big data for development?: Some examples 28

29 UN Global Pulse Global Post-2015 Twitter Conversation Searches approximately 500mn tweets daily across 16 key development topics (represented by about 25k keywords in multiple languages) 29

30 Google Dengue Trends ( Uses global search activity Follow-up to the successful (or not) Google Flu Trends ( but more on that later 30

31 That s cool, but who can get such work done? Answer: collaborative inter-disciplinary teams Data mining / computer science Information science Econometrics/ statistics Social science (economics, political science, sociology, anthropology, etc.) Domain expertise (e.g. urban & transport policy, etc.) 31

32 Spillovers from the big data phenomenon Greater emphases on data driven decision making Better tools for data extraction, analysis and visualization even for small data PDF to excel convertors e.g. Tabula Google Fusion Tables Data Driven Documents (D3) 32

33 Analytical challenges of working with big data Data provenance & data cleaning Representativeness Behavioral change Ground context Causation vs. correlation The role of small data for verification as well as for bootstrapping Transparency & replicability 33

34 Data provenance Data provenance is the process of tracing the pathways taken by data from the originating source through all the processes that may have mutated and/or replicated the data up until it is utilized for the analyses in question How relevant is it for big data analyses? Quite relevant though it may not be feasible to establish provenance to the extent desired by the scientific community E.g. when analyzing CDRs, some operators may have multiple records for forwarded calls. Not knowing this, can potentially affect social network analysis 34

35 Data cleaning Same steps apply for big data as for small data : First, verify that quantitative and categorical variables are coded as it should be Second, remove outliers. The methods though might have to be different E.g. when removing outliers, you have to use decision tree algorithms or heuristics such as 3-sigma from the mean, etc. 35

36 Is the data representative? Large volume may make sampling rate irrelevant but doesn t necessarily make the data representative Question: can you think of some representativeness issues in reference to mobile network big data? 36

37 Changes in behavior Once you know the process people will try to beat it E.g. Google page rank and Search Engine Optimization (SEO) Digitized online behavior can be subject to self-censorship and the creation of multiple personas E.g. your Facebook and Twitter posts can mirror how narcissistic you are Transactional data (by-product of doing something else e.g. making phone calls) is therefore the best form of behavioral data But even transactional data can be subject to behavioral changes Question: can you think of an example with respect to mobile network big data? 37

38 Ground context Knowing and understanding real world context is important, otherwise you might make false inferences Example from Sri Lanka: The majority of airtime reloads occur through scratch cards. Higher denomination cards are not easily available Question: Based on what you learnt earlier about how to differentiate between those form lower and higher income households, can you identify a potential problem when trying to develop poverty maps from mobile network big data? 38

39 Causation versus Correlation There are often more police in precincts with high crime, but that does not imply that increasing the number of police in a precinct would increase crime Source: Varian, H. R. (2013). Big Data: New Tricks for Econometrics. Available at Big Data analyses can ONLY reveal correlation NOT causation You can get to causal effect in a Big Data world by running experiments, but you and I cannot do that easily 39

40 But in some instances correlation can be enough to take actions E.g. Street Bump mapp in Boston Released by Boston City Hall for smartphones Once loaded on smartphones, uses the phone s accelerometer to detect potholes when users are driving, and then notifies City Hall of location. Certain changes in accelerometer readings correlates with passing over a pothole, but no guarantee of causal effect. Question: Why is correlation enough to take action in this case? 40

41 The role of traditional small data in verification and bootstrapping Verification: E.g. we need transport survey data to verify if the patterns of mobility discovered using mobile network big data meshes with real conditions Bootstrapping E.g. We can use mobile network big data from around the time of the census to build models to approximate socio-economic levels from just mobile network big data. This is then used to reverse engineer approximate census/ poverty maps in future when other data is not easily available. See Frias-Martinez, V., & Virseda, J. (2012). On the relationship between socio-economic factors and cell phone usage. In both cases, big data is complement and not a substitute for small data, except in some cases. 41

42 Transparency & replicability A lot of interesting big data sources are held by private entities The benefits of peer-review not yet easily available for much big data based research Less transparency and replicabilitymeans less chance of catching errors and improving the models 42

43 An illustration of the challenges using the example of Google Flu Trends (GFT) Since being launched successfully in 2008, subsequent research by others has found problems with how well GFT can predict incidence of Flu Problems: Original results based on bootstrapping with data from CDC, but there was no subsequent fine-tuning Over time more and more people started using Google to search for information on health issues, creating rebound effects post introduction of GFT Influenza-like illness rates did not necessarily correlate with actual influenza virus rates Original research article did not reveal the specific 45 search terms that were used to build the model 43

44 Analytical challenges of working with big data Data provenance & data cleaning Representativeness Behavioral change Ground context Causation vs. correlation The role of small data for verification as well as for bootstrapping Transparency & replicability For more discussion see my guest blog post at 44

45 Other challenges of trying to leverage big data for development Getting access to data Privacy 45

46 Getting access Some, such as Twitter data are accessible to all (with some volume limits) But for most private data sources, initially it will be through ad hoc means, though that is changing E.g. Orange Data for Development competition & Telecom Italia Big Data challenge See ml LIRNEasia and others working towards opening up the playing field for others 46

47 Privacy There are personal and collective privacy implications The difficulty is that: Informed consent is meaningless in a big data world People actually don t really know what their generalizable privacy needs or how their preferences might evolve Ways to deal with the privacy issue: Anonymization (with caveats) Different levels of disaggregation of shared data Evolution from a rights based approach to a harms based approach However more research still required 47

48 What you should have gained from today s lecture (hopefully ) Some understanding of what big data means An appreciation for some of the developmental opportunities from leveraging big data using different sources A brief overview of the spillover effects of the big data phenomenon An appreciation of some of the analytical and other challenges concerning the use of big data for development 48

49 Thank you 49