How big is big data? An introduc+on for developmental professionals and researchers

Transcription

1 How big is big data? An introduc+on for developmental professionals and researchers Sriganesh Lokanathan, LIRNEasia Young Scholar Seminar CPRsouth 2014 Maropeng, South Africa 7 September 2014 This work was carried out with the aid of a grant from the InternaHonal Development Research Centre, Canada and the Department for InternaHonal Development UK.

2 Yup, big data is huge and it will grow exponenhally In 2013 humans created 4.4 zetabytes of digihzed data 1 zetabye = bytes To put that in perspechve: That is as much data as contained in 202 trillion print issues of the Economist magazine If laid out each sheet from all the magazines side by side, it would cover the Earth 4.4 Hmes x202 trillion If we stored that data on 2- TB external hard drives it would require 2.3 billion of them Enough to take up the space of 1071 Great Pyramids x1071 Sources: Calculations by Lokanathan based on IDC (2014), Letouze (2014) 2

3 But the big data phenomenon is not just about the size 3

4 Big Data The Vs VELOCITY Rate of change of data VOLUME Size of data VARIETY Different forms of data VERACITY Uncertainty of data VALUE PotenDal value of data 4

5 What has facilitated the rise of the big data phenomenon? Vast drops in the cost of storing and retrieving informahon ExponenHal growth in computer power and memory data can reside in persistent memory instead of disk and tape Major improvements in techniques for performing machine learning and reasoning 5

6 Sources of big data AdministraHve data E.g. digihzed medical records, insurance records, tax records, etc. Commercial transachons E.g. Bank transachons, credit card purchases, supermarket purchases, online purchases, etc. Sensors and tracking devices E.g. road and traffic sensors, climate sensors, equipment & infrastructure sensors, mobile phones, satellite/ GPS devices, etc. Online achvihes/ social media E.g. online search achvity, online page views, blogs/ FB/ twiaer posts, online audio/ video/ images, etc. 6

7 Big data isn t necessarily a new area: the census is the original big data First recorded census occurred in Egypt (circa 3340 BC) But the tools didn t exist to fully exploit them The first tools invented for the 1890 US census It took 7 years to process the 1880 census data Herman Hollerith invents the tabulahng machine to process the 1890 census data Source: Adam Schuster 7

8 Today s Big data is tomorrow s data At its core, the big data phenomenon is really about data analyhcs, albeit at a scale that was not imaginable before 8

9 If we want comprehensive coverage of the populahon, what are the sources of big data in developing economies? AdministraHve data? E.g. digihzed medical records, insurance records, tax records, etc. Commercial transachons? E.g. Bank transachons, credit card purchases, supermarket purchases, online purchases, etc. Sensors and tracking devices? E.g. road and traffic sensors, climate sensors, equipment & infrastructure sensors, mobile phones, satellite/ GPS devices, etc. Online achvihes/ social media? E.g. online search achvity, online page views, blogs/ FB/ twiaer posts, online audio/ video/ images, etc. 9

10 Currently only mobile network big data has the widest possible populahon coverage 10

11 LIRNEasia s Big Data for Development (BD4D) Research LIRNEasia has negohated access to historical and anonymized telecom network big data from mulhple operators in Sri Lanka In the current research cycle we are conduchng exploratory research on answering a few social science queshons related to mobility and connectedness developing a framework with privacy and self- regulatory guidelines for the collechon, use and sharing of mobile phone data. hap://lirneasia.net/projects/bd4d/ 11

12 The data sets MulHple mobile operators in Sri Lanka have provided LIRNEasia access to 4 different types of meta- data: Call Detail Records (CDRs) Records of calls, SMS- es, Internet access AirHme recharge records Data sets do not include any Personally IdenHfiable InformaHon (PII). All phone numbers are anonymized and LIRNEasia does not maintain any mappings of idenhfiers to original phone numbers 12

13 What are some types of big data captured by mobile network operators? Call Detail Record (CDR) Records of all calls made and received by a person created mainly for the purposes of billing Similar records exist for all SMS- es sent and received as well as for all Internet sessions The Cell ID in turn has a lat- long posihon associated with it. 13

14 What are some types of big data captured by mobile network operators (contd.)? AirDme reload records Records of all airhme reloads performed by prepaid SIMs Each row corresponds to a record of one person s achvity: 14

15 How can we use mobile network big data for development? 15

16 We can understand the mobility paaerns of the populahon Low High Volume of people 16

17 We can understand land use characterishcs People leave digital traces when they use communicahon devices. Diurnal paaerns of user achvity in different locahons can be used to classify land use characterishcs Euclidean distance between two Hme series is used to cluster base stahons in an unsupervised manner using k- means algorithm 17

18 With even a simplishc clustering into just two regions we can clearly see the commercial areas Cluster 1 corresponds to more commercial area Cluster 2 corresponds to more residenhal (and mixed- used regions) 18

19 We can understand the geospahal distribuhon of the social networks in the country Each link represents the normalized volume of calls between two DSDs Divisional Secretariat Division (DSD) is a third level administrahve division; 331 in total in LK Low No. of calls High 19

20 We can understand community structures The social network is segregated such that overlapping connechons between communihes are minimized using a modularity approach For Sri Lanka, the ophmal number of communihes discovered by the algorithm was 11 20

21 How much do these communihes mesh with exishng administrahve boundaries? Southern and Northern provinces have the highest similarity to their respechve provincial boundaries. 14

22 How much do these communihes mesh with exishng administrahve boundaries (contd.)? Colombo district is clustered as a single community and Gampaha is merged with North Western Province 15

23 We can leverage the differenhal paaerns of airhme recharge. Average recharge amount Higher household income Lower household income Recharge frequency 23

24 to create poverty maps An example from Cote d Ivoir Source: Thoralf GuHerrez, GauHer Krings, Vincent D. Blondel, (2013). EvaluaHng socio- economic state of a country analyzing airhme credit and mobile phone datasets. Available at hap://arxiv.org/abs/ Average airtime recharge 24

25 What s the use of such work? Can bring Hmely evidence into the policy making process in developing economies 25

26 An example of Hmely policy- relevant evidence The image on the leq depicts relahve density of people in Colombo city and the surrounding regions at 1300 compared to 0000 (midnight the previous day) on a normal weekday. The yellow to red colors depict areas whose density has increased relahve to midnight. The blue color depicts areas whose density has decreased relahve to midnight (the darker the blue, the greater the loss in density). The clear areas are those where the overall density has not changed. 26

27 Our findings closely match results from expensive & infrequent transportahon surveys 27

28 Taking a broader view: the overall process of leveraging mobile network big data for development Mobile network big data (CDRs, Internet access usage, airhme recharge records) Construct behavioral variables (i) Mobility variables (ii) Social variables (iii) ConsumpHon variables Other data sources (i) Census data (ii) HIES data (iii) Survey maps (iv) TransportaHon schedules (v) ++++ Analytics (i) (ii) Insights Urban planning Crisis management & DRR (iii) Health monitoring & planning (iv) Socio- economic monitoring & planning 28

29 What are other potenhal data sources for big data for development?: Some examples 29

30 UN Global Pulse Global Post Twiaer ConversaHon hap://post2015.unglobalpulse.net/ Searches approximately 500mn tweets daily across 16 key development topics (represented by about 25k keywords in mulhple languages) 30

31 Google Dengue Trends (hap:// Uses global search achvity Follow- up to the successful (or not) Google Flu Trends (hap:// ), but more on that later 31

32 That s all cool, but who can get such work done? Answer: collaboradve inter- disciplinary teams Data mining / computer science InformaHon science Econometrics/ stahshcs Social science (economics, polihcal science, sociology, anthropology, etc.) Domain experhse (e.g. urban & transport policy, etc.) 32

33 Spillovers from the big data phenomenon Greater emphases on data driven decision making Beaer tools for data extrachon, analysis and visualizahon even for small data PDF to excel convertors e.g. Tabula hap://tabula.nerdpower.org/ Google Fusion Tables haps://support.google.com/fusiontables/answer/ VisualizaHon: Data Driven Documents (D3) hap:// InfoVis hap://philogb.github.io/jit/ 33

34 AnalyHcal challenges of working with big data Data provenance & data cleaning RepresentaHveness Behavioral change Ground context CausaHon vs. correlahon The role of small data for verificahon as well as for bootstrapping Transparency & replicability 34

35 Data provenance Data provenance is the process of tracing the pathways taken by data from the originahng source through all the processes that may have mutated and/or replicated the data up unhl it is uhlized for the analyses in queshon How relevant is it for big data analyses? Quite relevant though it may not be feasible to establish provenance to the extent desired by the scienhfic community E.g. when analyzing CDRs, some operators may have mulhple records for forwarded calls. Not knowing this, can potenhally affect social network analysis 35

36 Data cleaning Same steps apply for big data as for small data : First, verify that quanhtahve and categorical variables are coded as it should be Second, remove outliers. 36

37 Is the data representahve? Large volume may make sampling rate irrelevant but doesn t necessarily make the data representahve QuesHon: can you think of some representahveness issues in reference to mobile network big data? 37

38 Changes in behavior Once you know the process people will try to beat it E.g. Google page rank and Search Engine OpHmizaHon (SEO) DigiHzed online behavior can be subject to self- censorship and the creahon of mulhple personas E.g. your Facebook and Twiaer posts can mirror how narcissishc you are hap:// /University- study- finds- Facebook- Twiaer- fuel- narcissism- different- ways.html TransacHonal data (by- product of doing something else e.g. making phone calls) is therefore the best form of behavioral data But even transachonal data can be subject to behavioral changes QuesHon: can you think of an example with respect to mobile network big data? 38

39 Ground context Knowing and understanding real world context is important, otherwise you might make false inferences Example from Sri Lanka: The majority of airhme reloads occur through scratch cards. Higher denominahon cards are not easily available QuesHon: Based on what you learnt earlier about how to differenhate between those form lower and higher income households, can you idenhfy a potenhal problem when trying to develop poverty maps from mobile network big data? 39

40 CausaHon versus CorrelaHon There are oqen more police in precincts with high crime, but that does not imply that increasing the number of police in a precinct would increase crime Source: Varian, H. R. (2013). Big Data: New Tricks for Econometrics. Available at hap://people.ischool.berkeley.edu/~hal/papers/2013/ml.pdf Big Data analyses can ONLY reveal correlahon NOT causahon You can get to causal effect in a Big Data world by running experiments, but you and I cannot do that easily 40

41 But in some instances correlahon can be enough to take achons E.g. Street Bump mapp in Boston hap:// Released by Boston City Hall for smartphones Once loaded on smartphones, uses the phone s accelerometer to detect potholes when users are driving, and then nohfies City Hall of locahon. Certain changes in accelerometer readings correlates with passing over a pothole, but no guarantee of causal effect. QuesHon: Why is correlahon enough to take achon in this case? 41

42 The role of tradihonal small data in verificahon and bootstrapping VerificaHon: E.g. we need transport survey data to verify if the paaerns of mobility discovered using mobile network big data meshes with real condihons Bootstrapping E.g. We can use mobile network big data from around the Hme of the census to build models to approximate socio- economic levels from just mobile network big data. This is then used to reverse engineer approximate census/ poverty maps in future when other data is not easily available. See Frias- MarHnez, V., & Virseda, J. (2012). On the relahonship between socio- economic factors and cell phone usage. In both cases, big data is complement and not a subshtute for small data, except in some cases. 42

43 Transparency & replicability A lot of intereshng big data sources are held by private enhhes The benefits of peer- review not yet easily available for much big data based research Less transparency and replicability means less chance of catching errors and improving the models 43

44 An illustrahon of the challenges using the example of Google Flu Trends (GFT) Since being launched successfully in 2008, subsequent research by others has found problems with how well GFT can predict incidence of Flu Problems: Original results based on bootstrapping with data from CDC, but there was no subsequent fine- tuning Over Hme more and more people started using Google to search for informahon on health issues, creahng rebound effects post introduchon of GFT Influenza- like illness rates did not necessarily correlate with actual influenza virus rates Original research arhcle did not reveal the specific 45 search terms that were used to build the model 44

45 AnalyHcal challenges of working with big data Data provenance & data cleaning RepresentaHveness Behavioral change Ground context CausaHon vs. correlahon The role of small data for verificahon as well as for bootstrapping Transparency & replicability For more discussion on the above see my guest blog post at h6p:// challenges- big- data 45

46 Other challenges of trying to leverage big data for development Gezng access to data Privacy 46

47 Gezng access Some, such as Twiaer data are accessible to all (with some volume limits) But for most private data sources, inihally it will be through ad hoc means, though that is changing E.g. Orange Data for Development compehhon & Telecom Italia Big Data challenge See hap:// & hap:// LIRNEasia and others working towards opening up the playing field for others 47

48 Privacy There are personal and collechve privacy implicahons The difficulty is that: Informed consent is meaningless in a big data world People actually don t really know what their generalizable privacy needs are or how their preferences might evolve Ways to deal with the privacy issue: AnonymizaHon (with caveats) Different levels of disaggregahon of shared data EvoluHon from a rights based approach to a harms based approach Also see: hap://lirneasia.net/wp- content/uploads/2014/08/draq- guidelines- 2.2.pdf 48

49 What you should have hopefully gained from today s lecture Some understanding of what big data means An appreciahon for some of the developmental opportunihes from leveraging big data using different sources A brief overview of the spillover effects of the big data phenomenon An appreciahon of some of the analyhcal and other challenges concerning the use of big data for development 49

50 Thank you. 50