FluMapper: An Interactive CyberGIS Environment for Massive Location-based Social Media Data Analysis

Transcription

1 FluMapper: An Interactive CyberGIS Envirnment fr Massive Lcatin-based Scial Media Data Analysis Anand Padmanabhan 1,4, Shawen Wang 1,2,3,4, Gufeng Ca 1, MyungHwa Hwang 1, Yanli Zha 1, Zhenhua Zhang 1,5, Yizha Ga 1, Kiumars Sltani 1 1 CyberInfrastructure and Gespatial Infrmatin Labratry (CIGI) Department f Gegraphy and Gegraphic Infrmatin Science Schl f Earth, Sciety and Envirnment 2 Department f Cmputer Science 3 Department f Urban and Reginal Planning 4 Natinal Center fr Supercmputing Applicatins (NCSA) University f Illinis at Urbana-Champaign 5 Cmputer Netwrk Infrmatin Center, Chinese Academy f Sciences 1

2 Outline Mtivatins Backgrund FluMapper Spatial Data Handling Spatial Data Analysis Dem Cnclusins and Future wrk

3 Mtivatins Current influenza like illnesses (ILI) surveillances fcus n a certain spatitempral scale, usually at a state (r city) level and are typically updated weekly Early detectin and prgressin ILI are imprtant in public health research By expliting scial media data we can d much better Recent studies with significant scietal impacts have been cnducted with scial netwrking and media data E.g. predictins f the stck market (Bllen et al. 2011), disaster respnses (Gdchild and Glennn 2010) t infectius disease tracking (Signrini and Segre 2011, Wang et al. 2013) Dramatic increase in the vlume f spatial data available via scial media 2-4% f the 340M daily tweets sent via Twitter have GPS infrmatin available This number is grwing with increase in usage f lcatin-aware mbile devices 3

4 Backgrund Cnventinal appraches t disease surveillance use indicatrs like daily visits t physician and hspital visits, and labratry-test requests Thugh pssibly mre accurate, this type f infrmatin is difficult t cllect and hence there is a ptential time lag between an utbreak and when it gets reprted Alternative appraches t disease surveillance like search (Ginsberg et al. 2008), nline news and dcuments (Keller et al ) and lcatin-based scial media (LBSMD) data (Wang et al. 2013) have been investigated Use f LBSMD in disease surveillance is increasing (Achrekar et al. 2011, Chen et al. 2010, Cllier et al.2012, Gmide et al. 2011, Quincey et al. 2010, Signrini et al. 2011,Wang et al. 2013) 4

5 Advantages Using LBSMD Rich infrmatin abut varius scial themes and captures the dynamic prcess f hw humans interact ver time and space (Tsu et al. 2013) Unlike traditinal data, like frm surveys and clinical reprts, that prduce retrspective infrmatin and are cst prhibitive, LBSMD can be cllected in near real-time with much less cst (Signrini et al. 2011) Individual-level spatial and tempral bservatins allw explratry analysis f spatitempral dynamics f disease prcesses n varying scales (Wang et al. 2013) Far less restrictin n use f LBSMD cmpared t clinical r survey data Challenges Data sizes quickly grw beynd the capabilities f cnventinal gegraphic infrmatin systems (GIS) (Wang et al. 2013) Data are in unstructured textual frm s intensive data mdeling and prcessing is required t transfrm LBSMD int structured spatial data entities (e.g., pint patterns and trajectries) n which spatitempral analysis culd be applied (Li et al. 2013) Sampled natured f LBSMD data raises questins f quality and statistical validity 5

6 CyberGIS Sftware Ecsystem CyberGIS Scial Media Analytics CyberGIS Gateway and Applicatins CyberGIS Tlkit GISlve Middleware Viewshed Analysis Spatially Explicit Agent-Based Mdeling f Disease Spread FluMapper 6

7 CyberGIS Gateway Online cllabrative gespatial prblem slving envirnment Enables easy access t CyberGIS analytics and data surces Prvides transparent access t a rich set f cyberinfrastructure envirnments Represents a brad apprach t CyberGIS Widely accessible 7

8 FluMapper Gals Prvide early ILI activity detectin t public health researchers and practitiners Efficiently manage the large vlumes f LBSMD Capture the dynamics f flu risk acrss multiple spatitempral scales Prvide ILI risk maps frm natinal scale t very fine lcal scales Identify mbility patterns f ppulatin Aggregate individual user trajectry acrss different spatial and tempral scales t generate flws Capture travel patterns acrss the cuntry, reginal, state, city and lcal levels Facilitate interactive and explratry analysis f LBSMD Update ILI activity in near real-time fr the cnterminus United States Prvide an integrated view f flu risk indicatrs and mbility f users 8

9 FluMapper Cmpnents Data cllectin and prcessing Cllects, prcesses and stres data frm Twitter in near real time Efficient and scalable services t query raw and derived data develped Spatitempral data mdel Prvides aggregated data and statistics at multiple scales fr efficient spatial query At the finest scale, the cnterminus United States is represented as a unifrm grid with cell size f 30 arc secnds Explratry spatial data analysis Kernel density estimatin (KDE), a spatial cluster detectin technique, is used t detect unusual cncentratin f flu events Mnte-Carl simulatins with randm labeling are cnducted t imprve ur cnfidence in analysis results Flw mapping analysis Single-surce flw mapping is applied fr mapping the travel pattern f ppulatin frm r t a specific area 9

10 Overall Architecture After (Wang et al. 2013) 10

11 Data Cllectin and Prcessing Prvides near real-time access t data frm Twitter Data frm Twitter cllected using streaming applicatin prgram interface (API) Transfrms tweets int structured spatial data entities Facilitates multiscale explratry spatitempral analytical methds f flu risks and the related ppulatin mvements Interactive analysis uses a 7-day sliding windw Prvides spatitempral surveillance f flu by prcessing tweets cntinuusly and aggregating summary flw data in a hierarchical multi-scale structure Prvides scalable service interface t access the massive data stream MngDB, a scalable and pen surce NSQL database used Set f distributed databases fr managing spatial data entities f tweets and flws 11

12 Data Pipeline Architecture 12

13 Crawler Data Pipeline Cmpnents Use Twitter s streaming API ~2.5M tweets daily (Data has been cllected fr a year) A spatial filter (n keywrd filter) f Nrth America is applied when data is cllected Tweet imprter Reads in raw tweets and cnverts them int spatial events Uses text mining techniques t identify flu related tweets (Signrini et al. 2011) Cube generatr Generates trajectries f individual users Aggregates these trajectries int flws at multiple spatitempral scale Cube cnsisted f ten layers f spatial grids, and at the finest level cntained a spatial grid f cells f abut 30 arc-secnds ( 1 km) width Data filter service Prvides spati-tempral query capabilities t the flu and flw databases 13

14 Data Characteristics Daily vlumes f tweets (May 23 ~ June 5, 2013) Spatial distributin f Twitter users (May 31~ June 6, 2013) 14

15 Spatitempral Data Mdel Spatitempral data cube is develped t supprt efficient query f aggregated statistics Data cube decmpses the spatitempral space int a lattice f multi-scale, hierarchical cubids Prvides multiple scales f spatial indices fr fast and efficient spatial query Given a regin, statistics such as the number f flu ccurrences, and number f trajectries traveling ut and in frm this regin are efficiently retrieved

16 Data in Spatitempral Cube Specific measures/statistics f ge-lcate tweets are defined and cmputed fr each cubid Fr each cubid at multiple scales, the fllwing measures are cmputed recursively: Number f twitters Number f tweets Number f flu tweets Number f trips traveling ut f a cubid Number f trips traveling t the cubid Fr each pair f cubids at multiple scales, the fllwing measures are cmputed recursively: Number f trips traveling frm a cubid t the ther. Number f flu related trips traveling frm a cubid t the ther At the finest scale, the cnterminus United States is represented as a unifrm grid with cell size f 30 arc secnds

17 An Example Spatitempral Data Cube Sizes f a cllectin f spatitempral data cube DB (May 23 ~ June 5, 2013)

18 Input Explratry Spatial Data Analysis Individual tweets in the Twitter feeds are indexed by bth space and time Flu-related tweets represent infectin case, while all tweets tgether frm the backgrund ppulatin Result Spatitempral maps exhibiting flu risk and unusual cncentratin f flu events Technique Kernel density estimatin (KDE), an explratry spatial data analysis technique t identify spatial clusters, applied t detect cncentratin f flu events (Shi. 2010) Adaptive bandwidth is applied t accunt fr inhmgeneus backgrund ppulatin Mnte-Carl simulatins with randm labeling are cnducted t imprve ur cnfidence in analysis results Scale is a crucial factr that influences explratry spatial data analysis results and s we cnduct ur analysis at multiple scales t discver spatial patterns Cde develped t run n GPUs (Graphical Prcessing Units) and executed n XSEDE 18

19 Adaptive Kernel Density Estimatin Estimate risk surface by using a kernel whse bandwidth is adapted t cnstant ppulatin pints (Shi 2010) Calculate adaptive bandwidth f each pixel k nearest neighbr search (KNN search) Estimate risk surface using KDE estimatr Mnte Carl simulatin Cmputatinally intensive Spatially srt pints int tiles Search neighbring tiles Partial srt tiles t find k neighbrs O(NMk) N -the number f cells in risk surface M -the number f pints k - the kth neighbr t be searched

20 Scalability Challenges and Spatial Strategies Cmpute nde CPUs GPUs Cmpute nde CPUs GPUs Challenges Spatial Strategies Lad imbalance Carse-scale spatial cmputatinal dmain decmpsitin parallel threads glbal memry GPU Intensive memry access Fine-scale spatial data dmain decmpsitin

21 Addressing Lad Imbalance Prblem Traditinal apprach t dmain decmpsitin is based n spatial data dmain withut cnsidering cmputatin Causes lad imbalance Slutin Cnstruct spatial cmputatinal dmain (Wang et al. 2009) Perfrm carse-scale dmain decmpsitin n spatial cmputatinal dmain Decmpsitin n spatial cmputatinal dmain, guided by cmputatinal intensity Prperties f data and peratins in spatial dmain transfrmed t cmputatinal intensity in spatial cmputatinal dmain A transfrm functin defined fr a particular spatial analysis Accunt fr hetergeneity f underlying cmputing architecture Grup decmpsed dmain int parallel tasks Achieve hmgeneity between these parallel tasks using the cmputatinal intensity values

22 Carse-scale Decmpsitin fr KDE Transfrm Functin fr KDE As the number/density f pints in a cell and its intermediate neighbrs decreases, the cmputatin intensity f a task assciated with that cell increases Carse decmpsitin uses adaptive quadtree and applies transfrm functin t estimate cmputatinal intensity Using a space-filling curve tiles are gruped tgether t frm parallel tasks The gal is t minimize the difference in cmputatinal intensity values amng the parallel tasks Cmputatinal intensity value GPU1 GPU2 GPU GPU4 spatial cmputatinal dmain

23 Evaluating Perfrmance f Carse-scale Spatial Cmputatinal Dmain Decmpsitin Regular decmpsitin Task number Our strategy k=500, Keeneland cluster 16GPUs (regular) 16GPUs(ur strategy) 1GPUs Ttal 168sec 65sec 890sec Task1 166sec 43sec Task2 30sec 40sec Task3 50sec 32sec Task4 13sec 49sec Task5 26sec 63sec Task6 103sec 40sec Task7 11sec 41sec Task8 86sec 14sec Task9 26sec 36sec Task10 13sec 15sec Task11 21sec 39sec Task12 7sec 45sec Task13 13sec 39sec Task14 35sec 28sec Task15 13sec 48sec Task16 30sec 43sec

24 Addressing Intensive Memry Access Prblem Many threads directly access t lng-latency glbal memry Slutin Fine-scale spatial dmain decmpsitin applied Explits spatial prperties Spatial lcality - data culd be shared amng neighbrs Spatial dependency - dependent data can be identified by scpe f spatial peratins, e.g., neighbr scpe Decmpse space int fine-scale tiles If tile size is t large, the it may invlve many irrelevant data If tile size is t small, then it may reduce data reuse Tile size is als cnstraint by hardware resurces Tile size is usually tuned experimentally Spatially srt data int tiles Grup memry access f parallel threads in a tile by calescing them int a thread blck Search, pre-fetch and cache dependent data f a thread blck int shared memry 24

25 Fine-scale Decmpsitin fr KDE cache Shared memry utput thread1 thread2 thread3 thread4 A thread blck Search dependent data cmputatin Threads directly access t fast shared memry while cmputatin and reuse data Apprach shifts access frm slw glbal memry t fast shared memry Shared memry Glbal memry input

26 Evaluating Perfrmance f Fine-scale Spatial Data Dmain Decmpsitin Cmputing time(secnds) Tile size k=500, keeneland cluster, 1GPUs

27 Flw Mapping Analysis Mvement data visualizatin and analysis methd Each edge represents a mvement (flw) between a pair f gegraphical regins and thickness f each edge is crrespnding t the flw magnitude n this edge Flw mapping in FluMapper Single-surce flw mapping is applied fr mapping the travel pattern f ppulatin frm r t a specific area Relies n spatitempral data cube mdel Individual trajectries are identified frm Twitter feeds Each trajectry prvides t the spatitempral ftprint f a Twitter user Trajectries are aggregated by the spatitempral data cube at multiple scales Supprts visualizatin multiple scales f Twitter flu users mbility Flw infrmatin culd help cntextualize disease spread within and between regins 27

28 Flw Mapping: Challenges and Slutins Visualizing flws cntained in large vlumes f LBSMD data are bth cmpute- and data- intensive CyberGIS envirnment reslves the cmputatinal challenges Visualizing flw between a large number f rigindestinatin can lead t cluttered display Spiral tree-based algrithm adpted t minimize visual clutter (Verbeek et al ) Prperties f spiral tree avid intersectins Remaining Prblems Des nt cnsider gegraphy f peple mvement Peple usually mve n the rads, r by airplane Difficult t fully capture the reginal activity 28

29 Flw Mapping Cnsidering Gegraphy Cnsider grund and air based mvements separately Slutins Grund mvement uses ruting service Air rutes use spiral-tree apprach Online Ruting Services Ggle Directin API Mapquest Directin API with OSM data Drawbacks Call times and frequency restrictin Perfrmance challenges wing t thrttling, cnnectin stability and cmpetitin with ther applicatins Lcally deplyed ruting service OSRM(Open Surce Ruting Machine) Uses OSM(Open Street Map) data Advantages N restrictin Cnfigurable and adaptable Lcally deplyed ruting service was chsen 29

30 Architecture

31 Perfrmance Evaluatin Surce Place MapQuest(ms) (Psting requests with multi-threads in Client side) Lcal OSRM(ms) (Psting requests with multithreads in Client side) Lcal OSRM(ms) (with multi-threads in server side) Texas Flrida Kansas New Yrk Califrnia Chicag

32 Flw Mapping M t N Visualize all the activities (all surce t all target) in the bservatin windw Tackles large data size, arund 500k pints at finest level Applies visualizatin strategies t reduce clutter Majr cmpnents Backend: (a) reads data frm MngDB database; (b) transfrms data int a graph frmat; and (c) identifies imprtant ndes in each regin Cmputatinally intensive step hence it is pre-cmputed Sevlet: (a) prcesses request frm ur web frntend; (b) queries the result frm the backend prcessing; (c) runs Frce Directed Edge Bundling algrithm n the query utput t prduce a visually desirable result This cmpnent prcesses request in real time and prduces a weighted directed graph Web frntend: Use weighted graph returned by the servlet t categrize graph edges and clr them 32

33 Dem 33

34 Summary Prviding early ILI activity detectin t public health researchers and practitiners is crucial LBSMD are well suited t prvide insights int this issue FluMapper handles big data effectively Spatitempral data mdel supprts efficient spatial queries Data prcessing pipeline handle large vlumes f LBSMD FluMapper prvides interactive spatial analysis n LBSMD Kernel density estimatin is used t identify unusual risk pattens Flw mapping prvides additinal explratry tl t cntextualize disease spread patterns CyberGIS prvides a suitable envirnment fr reslving cmputatinal challenges f FluMapper CyberGIS integrates cmputatin, data and cmmunicatin technlgies frm XSEDE t reslve cmputatinal challanges 34

35 Future Wrk Expand the spatial extent f FluMapper t Nrth America This will test the scalability f the data pipeline and query efficiency Studies t cmpare FluMapper results with thse frm CDC and Ggle Flu Trends are underway Imprve multi-surce t multi-destinatin flw mapping perfrmance Map-reduce apprach being investigated 35

36 Acknwledgements Natinal Science Fundatin BCS OCI OCI TeraGrid/XSEDE SES Clleagues 36

37 Thank Yu Questins? 37