GISolve-FluMapper for Interactive Spatial Analysis of Streaming Social Media Data

Transcription

1 GISlve-FluMapper fr Interactive Spatial Analysis f Streaming Scial Media Data Anand Padmanabhan 1,4, Shawen Wang 1,2,3,4, Gufeng Ca 6, MyungHwa Hwang 1, Yanli Zha 1, Zhenhua Zhang 1,5, Yizha Ga 1, Kiumars Sltani 1 1 CyberInfrastructure and Gespatial Infrmatin Labratry (CIGI) Department f Gegraphy and Gegraphic Infrmatin Science Schl f Earth, Sciety and Envirnment 2 Department f Cmputer Science 3 Department f Urban and Reginal Planning 4 Natinal Center fr Supercmputing Applicatins (NCSA) University f Illinis at Urbana-Champaign 5 Cmputer Netwrk Infrmatin Center, Chinese Academy f Sciences 6 Texas Tech University 1

2 Mtivatins Cnventinal appraches t disease surveillance use indicatrs like physician hspital visits, and labratry-test Influenza like illnesses (ILI) surveillances are usually cnducted n fixed spatitempral scale Typically updated weekly fr regins/states/cities Early detectin and prgressin ILI are imprtant in public health research Scial media data culd prvide an additinal indicatr Recent studies with significant scietal impacts have been cnducted with scial netwrking and media data E.g. predictins f the stck market (Bllen et al. 2011), disaster respnses (Gdchild and Glennn 2010) t infectius disease tracking (Signrini and Segre 2011, Wang et al. 2013) Dramatic increase in the vlume f spatial data available via scial media 2-4% f the 340M daily tweets sent via Twitter have GPS infrmatin available This number is grwing with increase in usage f lcatin-aware mbile devices 2

3 Using Lcatin Based Scial Media Data Advantages Cntains rich and dynamic infrmatin ver space and time abut varius scial themes (Tsu et al. 2013) Cst efficient and can be cllected in near real-time (Signrini et al. 2011) Individual-level bservatins allw explratry analysis f dynamics n varying spatitempral scales (Wang et al. 2013) Far less restrictin n use f lcatin based scial media Data (LBSMD) cmpared t clinical r survey data Challenges Data sizes quickly grw beynd the capabilities f cnventinal gegraphic infrmatin systems (GIS) (Wang et al. 2013) Data are in unstructured textual frm s intensive data mdeling and prcessing is required t transfrm LBSMD int structured spatial data entities (e.g., pint patterns and trajectries) n which spatitempral analysis culd be applied (Li et al. 2013) Sampled natured f LBSMD data raises questins f quality and statistical validity 3

4 FluMapper Gals Prvide an explratry tl t public health researchers fr detect ILI activity early as indicated in scial media data Efficiently manage the large vlumes f LBSMD Capture the dynamics f flu risk acrss multiple spatitempral scales Prvide ILI risk maps frm natinal scale t very fine lcal scales Identify mvement patterns f ppulatin Aggregate individual user trajectry acrss different spatial and tempral scales t generate flws Capture travel patterns acrss the cuntry, reginal, state, city and lcal levels Facilitate interactive and explratry analysis f LBSMD Update ILI activity in near real-time fr the cnterminus United States Prvide an integrated view f flu risk indicatrs and mbility f users 4

5 CyberGIS Sftware Ecsystem CyberGIS Scial Media Analytics CyberGIS Gateway and Applicatins CyberGIS Tlkit GISlve Middleware Viewshed Analysis Spatially Explicit Agent-Based Mdeling f Disease Spread FluMapper 5

6 FluMapper Architecture After (Wang et al. 2013) 6

7 Data handling FluMapper Cmpnents Data cllectin and prcessing Cllects, prcesses and stres data frm Twitter in near real time Efficient and scalable services t query raw and derived data develped Spatitempral data mdel Spatitempral data cube is prvides aggregated data and statistics at multiple scales fr efficient spatial query At the finest scale, Nrth America is represented as a unifrm grid with cell size f 30 arc secnds Spatial Analysis Explratry spatial data analysis Kernel density estimatin (KDE), a spatial cluster detectin technique, is used Mnte-Carl simulatins with randm labeling is applied Flw mapping analysis Scalable interactive multi-scale visualizatin f mvement pattern is massive scial media data was develped 7

8 Data Handling Prvides near real-time access t data frm Twitter Data frm Twitter cllected using streaming applicatin prgram interface (API) Applies techniques like supprt vectr machine (SVM) and keywrd filtering t identify flu-related tweets A 7-day sliding windw data is prvided fr interactive analysis GISlve middleware is enhanced t prvide scalable service interface t access the massive data stream MngDB, a scalable and pen surce NSQL database used Set f distributed databases manage spatial data entities Spatitempral surveillance f flu is achieved by prcessing tweets cntinuusly and aggregating summary flw data in a hierarchical multi-scale structure using spatitempral data cube 8

9 Data Handling - Architecture 9

10 Crawler Data Handling - Cmpnents Use Twitter s streaming API ~2.5M tweets daily (Data has been cllected fr a year) A spatial filter (n keywrd filter) f Nrth America is applied when data is cllected Tweet imprter Reads in raw tweets and cnverts them int spatial events Uses text mining techniques t identify flu related tweets (Signrini et al. 2011) Cube generatr Generates trajectries f individual users Aggregates these trajectries int flws at multiple spatitempral scale Cube cnsisted f ten layers f spatial grids, and at the finest level cntained a spatial grid f cells f abut 30 arc-secnds ( 1 km) width Data filter service Prvides spatitempral query capabilities t the flu and flw databases 10

11 Spatitempral Data Mdel Spatitempral data cube is develped t supprt efficient query f aggregated statistics Data cube decmpses the spatitempral space int a lattice f multi-scale, hierarchical cubids Prvides multiple scales f spatial indices fr fast and efficient spatial query Given a regin, statistics such as the number f flu ccurrences, and number f trajectries traveling ut and in frm this regin are efficiently retrieved

12 Spatial Analysis - Flw Mapping Scalable interactive visualizatin f mvement patterns in massive streaming scial media data was achieved Implemented tw independent but cmplementary appraches that shw flws t all destinatin frm (a) single-surce, and (b) all-surces (MvePattern) Multi-scale visualizatin f massive graphs cnducted A Hadp-based strategy fr data-intensive prcessing applied An adaptive and interactive user interface develped See tw student psters fr mre-details Scalable GISlve data services are used t efficiently query the data in spatitempral data cube and the mngdb database 12

13 MvePattern Architecture

14 Explratry Spatial Data Analysis Kernel density estimatin (KDE) is applied t detect cncentratin f flu events Adaptive bandwidth is applied t accunt fr inhmgeneus backgrund Mnte-Carl simulatins with randm labeling are cnducted (Shi. 2010) Spatitempral maps shwing the flu risk prduced Input are individual tweets in the Twitter feeds indexed by bth space and time Flu-related tweets represent infectin case, with all tweets frming the backgrund Analysis is cnducted at multiple spatial scales Might help t amelirate the effect f mdifiable areal unit prblem Cde develped t run n GPUs (Graphical Prcessing Units) and executed n XSEDE GISlve Open Service API used t cnduct the analysis 14

15 What was accmplished? Dem 15

16 GISlve Middleware Web Brwser Desktp Sftware Gateway GISlve Middleware Open Service API Service Infrastructure Analysis Data Cmputatinal Intensity Analysis Visualizatin Integratin Services Cmputatin Management Build & Test CI Resurce Management Cyberinfrastructure High Perfrmance High Thrughput Cluds

17 GISlve Open Service API Defines a set f REST Web service interfaces fr CyberGIS authenticatin, applicatin integratin, and cyberinfrastructure-based cmputatin Manages cmplexity f CyberGIS sftware integratin and cyberinfrastructure-based cmputatinal wrkflws Supprts brad access t integrated CyberGIS applicatins, data, and services Facilitates sftware integratin prcesses Client cdes written in multiple prgramming languages, including Java, Perl, PHP, and Pythn prvided 17

18 GISlve Open Service API (Cnt.) Security API: prvides an easy-t-use security mechanism fr CyberGIS sftware integratin, prgramming access t CyberGIS services, and cyberinfrastructure resurces Integratin API: prvides a service riented apprach fr CyberGIS sftware registratin, cnfiguratin, build & test, and publicatin & subscriptin Cmputatin API: prvides an easy-t-use interface fr cyberinfrastructure-based gespatial cmputatin 18

19 Cncluding Summary By leveraging the scalable and easy-t-use Open Service API prvided by GISlve middleware, CyberGIS sftware envirnment is able t reslve the cmpute and data challenges frm massive streaming scial media data CyberGIS integrates cmputatin, data and cmmunicatin technlgies frm XSEDE t reslve cmputatinal challenges FluMapper effectively handles big data and supprts interactive spatial analysis n it Prviding early ILI activity detectin t public health researchers and practitiners is crucial The techniques implemented in FluMapper are generalizable t ther prblems (e.g. disaster respnse) where spatial analysis f lcatin based scial media data culd prvide valuable insights 19

20 Acknwledgements Natinal Science Fundatin BCS OCI OCI TeraGrid/XSEDE SES Clleagues 20

21 Thank Yu Questins? 21