A data revolution for development? Giulio Quaggiotto UN Global Pulse Lab Manager,

Transcription

1 A data revolution for development? Giulio Quaggiotto UN Global Pulse Lab Manager,

2 The promise The challenge New data as a practice

3

4

5

6

7 A radically different view

8 An example in action

9 The future is already here, it is just not very evenly distributed

10

11

12

13 4 ways to data innovation 1. Funding and investment for national statistical capacity, particularly in developing countries. 2. Exploring new data sources, including those sourced from individual citizens. 3. Harnessing advanced technologies, like visualization tools that make data more understandable. 4. Liberating data to unleash the analytical creativity of users and hold policymakers accountable. U.N. Deputy Secretary-General Jan Eliasson

14 The challenge

15 New data as a practice Those who have done it Dev t sector Those who talk about it

16

17 Quiz time

18 GLOBAL PULSE: A NETWORK OF LABS Pulse Lab NYC Est Pulse Lab Kampala Est Pulse Lab Jakarta Est. 2012

19 New data as a practice

20 BIG DATA FOR DEVELOPMENT: 3 OPPORTUNITIES 1. Situational Awareness: Real-time trend analysis of population activities and dynamics can inform the design and targeting of programmes and policies. 1. Early Warning: Predictive analysis and detection of anomalous events allows rapid response to crises. 1. Programme Evaluation: Real-time feedback from citizens, and measurement of behavior change, allows for adaptive course corrections in programmes and policies.

21

22 Types of data Example data sources Global Pulse works with: Social media data (blogs, forums, social media streams) Mobile network data (CDRs, top-ups) Radio feeds News media content Online search Postal data GPS data We gain access to this type of data through partnerships with private sector or academia.

23

24

25

26

27

28 People tweet about immunization People share informative information on immunization and vaccinations with their connected friends Types No. of tweets Contents 1453 Is it dangerous to have fever, swelling and pain after vaccines? Shared Links 962 Is it true MMR vaccine can lead to autism in children? 800+ MoH launched Pentavalent vaccine and booster Retweets UNICEF reports that 30K-40K children are infected by measles every year in Indonesia. Fever, the initial symptom of measles, will increase within the first five days and then skin rash starts. Polio is a contagious disease, theoretically, which can infect all ages but children are most vulnerable.

29 People express opinions Number of tweets per day Reports in Media which prompts spikes in tweets [2013/12/01] Debates about assurance of halal products. [2013/12/03] Uncertainty whether some drugs contain pig substance [2013/12/06] MoH starts consultations related to halal certification. [2013/12/07] Debates over halal certificates of food. [2013/12/12] Confirmation that some drugs and vaccines may contain haram substance. [2013/12/12] MUI urges pharmacologists to replace haram process.

30 Situational awareness There are some autism cases after MMR vaccine A baby suddenly died after vaccine Is it dangerous to have fever, swelling, pain, after vaccine? China is investigating death cases of babies 0 June 2012 Oct 2012 Apr 2013 Dec 2013 Rank Autism (213) Death(1030) Fever (1498) Death (224) 2 Death (5) Fever (14) Swelling (1494) Fever (3) 3 Sick (4) Sick (4) Pain (1491) Crying (1) 4 Fever (2) Crying (3) Autism (1011) Autism (1) 5 Crying (1) Fever (3) Fever (4) -

31 Early warning and rapid response Early warning Rapid response with actionable plan Detect people concerned about death after vaccine from Twitter Disseminate correct information through Twitter via @blogdoktor Number of tweets of death

32 Finding extracts Gender in the workplace Workflow Feasibility Study Results Topic Brainstorming Promising topics -Permission to work -Perception: appropriateness of work -Discriminatory job requirement -Double burdens of working women Less promising topics -Cost to access employment -Lack of skills or education Taxonomy Design Feasibility Analysis Enough tweet volume AND Sufficient relevant tweets Enough tweet volume OR Sufficient relevant tweets

33 Finding extracts Nowcasting Food Prices and Understanding Coping Mechanisms Project Workflow Results with Initial Methodologies Data Collection & Refinement Nowcasting food prices Correlation study between (a) Real-word commodity price and (b) Commodity price from Twitter Understanding coping mechanism Pattern discovery from (a) Commodity price changes and (b) Consumption patterns Information Extraction Correlation Study We confirm that people discuss commodity price in social media and test the feasibility of extracted information. Cooking oil is a commodity which directly affects people s life because it is a basic commodity in Indonesia, differently from condensed milk.

34 Sinabung Eruption (15 th Sep, 2013) Infographics Location : Karo regency, North Sumatra Elevation : 2,460 m above sea level Victims : BNPB (Indonesian National Board of Disaster Management) reported 15 people died, and more than 30,000 people evacuated Volume Dynamics from Twitter Period: 14/9/2013 and 10/2/2014 Total Twitter Posts: 151,448 Relevant Posts: 117,436 (78%) More than 10K tweets at the first eruption

35 Visualizing Displacement Due to Floods through Mobile Data Partners: WFP, Govt. of Mexico, Univ. of Madrid, Telefonica Project: Visual analytics to support improved targeting of humanitarian assistance during emergencies

36

37 CDRs population estimate vs census - state of Tabasco, Mexico Source: Telefonica

38 A real-time map of poverty in Cote d Ivoire? a) Abidjan b) Liberian border c) Roads to Mali and Burkina Faso d) Road to Ghana Ref: arxiv.org/abs/ : Evaluating Socio-Economic State Of A Country Analyzing Airtime Credit And Mobile Phone

39

40 Luminosity as a proxy for GDP output Chen & Nordhaus, Using luminosity data as a proxy for economics statistics, 2011

41 Understanding labour market flows Source: Using social media to measure labour market flows, March 2014

42 M ashing big data with sensors

43

44 and citizen generated data

45 Evaluating policies real time? A mobility index to evaluate H1N1 response in Mexico City Telefonica Research, 2011 (

46 Real time evaluation of advocacy Finding extracts

47 Big postal data 1+ billion records per year Real-time scans since early 2010 Traffic btw 150 countries: Letters Packages MoneyGram International Postal Traffic: Daily Weight (kg) letter-post parcel-post EMS Letter-Post MA Letter-Post Trend Parcel Post MA Parcel Post Trend Express MA Express Trend

48

49 Making it happen: New data as a practice

50 New data as a practice Those who have done it Dev t & Gov t Those who talk about it

51

52 Anatomy of a big data project Questions Data dive Prototype Pilot Real time tools

53 Project portfolio Category Status Names 1. Social media for social protection Active 2. Social media to understand public perception of immunization 3. Signals of discrimination in the workplace 4. Nowcasting food prices and understanding coping mechanisms Research projects Exploration Ad-hoc 5. Mapping socio economic vulnerability 6. Maternal health 7. Disaster response/resilience 8. Universal heath coverage/public service monitoring 9. Deforestation 10. Providing Real-Time Insights on Indonesian Post2015 Priorities

54 Skillset of a data team

55

56 Big Data Access Data Mining & Analysis Technologies Leveraging Partnerships Twitter (global, 500 million messages/day) Orange France Telecom (Ivory Coast, Senegal) Telenor (Bangladesh mobile money data) Telefonica (Mexico, Guatemala) MTN (Uganda) Real Impact (Cote d Ivoire, Rwanda, Zambia) Universal Postal Union (global postal flow data) Amazon Web Services (supercomputing) DataSift (data filtering) SAS (analytics & data visualization) Crimson Hexagon (data analysis) Data Science Expertise Université catholique de Louvain (call records analysis) Institut des Systèmes Complexes de Paris Ile-de- France (news media mining & filtering) Universidad Politécnica de Madrid (call records analysis) Stockholm University (research fellow) Karolinska Institutet (call records analysis) University of Sheffield (speech-to-text tools) Microsoft Research (social media analysis)

57 Infrastructure example: Geolocaton Augmentation UNGP Code + Plain tweets AWS S3 Plain tweets Hadoop MapReduce HDFS / Impala GeoNames database Geolocated tweets

58

59

60

61

62 Easy-to-use tools for real-time awareness

63

64 Working with us Horizon scanning Trainings/capacity building Secondments & residencies Joint prototyping Full research project

65

66 Giulio Quaggiotto UN Global Pulse Lab Manager,

67 3 roles for NSOs and big data 1. 3 rd party to certify statistical quality of new sources 2. Issue statistical best practices in the use of nontraditional sources and the mining of big data 1. Use non-traditional sources to augment (and perhaps replace) official series Source: Andrew Wyckoff, OECD

68 4 big data illusions 1.uncannily accurate results 2.N=all statistical sampling is obsolete 3.causation is passee 4. scientific or statistical models aren t needed the end of theory Source: Tim Harford, FT

69 Big data technologies Big data requires different technologies and infrastructure! Lots inter-connected computers for data storage Distributed processing for analytics One of Google data center (source:

70 Big data applications Private Sector Item recommendation by Amazon Movie recommendation by Netflix Friend recommendation by Facebook Predicting loss of customers by mobile phone companies Optimizing Human Resources/hiring practices Public Policy Tax fraud detection Twitter earthquake monitoring Understanding perceptions and implications of policy decisions Now-casting food prices and inflation rates Big data applications already nearby us!

71 Data science Data science: a new discipline to analyze big data Data scientist: a practitioner of data science with various skills to analyze big data to find patterns, to generate insights, and to visualize results for a communication with others