Big Data in Pictures: Data Visualization

Transcription

1 Big Data in Pictures: Data Visualization Huamin Qu Hong Kong University of Science and Technology

2 What is data visualization? Data visualization is the creation and study of the visual representation of data - wiki Input: data Output: visual form Goal: insight

3 Why visualization?

4 Anscombe s Quartet: Four datasets

5 Anscombe s Quartet: Statistics

6 Anscombe s Quartet: Visualizations

7 Bible

8 GapMinder

9 GapMinder

10 Data Visualization Vis is hot & cool Vis is young What is vis research? What are we doing at HKUST?

11 WSJ

12

13

14

15

16

17

18 Obama s big data plan A $2 million award for a research training group in big data that will support training for undergraduate and graduate students and postdoctoral fellows using novel statistical, graphical, and visualization techniques to study complex data.

19 Tableau

20

21

22

23 Data Visualization Vis is hot & cool Vis is young What is vis research? What are we doing at HKUST?

24 Subfields Scientific Visualization (SciVis) Spatial data Information Visualization (InfoVis) Abstract data Visual Analytics (VAST) Analytical reasoning

25 VIS Conferences VAST (Visual Analytics Science and Technology) InfoVis (Information Visualization) SciVis (Scientific Visualization)

26

27

28

29

30 VIS - Infographics Infographics is static Visualization is interactive

31 StoryLine

32 StoryLine

33

34 VIS - HCI Visualization deals with data HCI deals with everything involving human & computer interaction

35 VIS - Data Mining Data mining focuses more on automatic algorithms Visualization keeps human in the loop and focuses more on interactive analysis

36 VA and Data Mining Data Analysis let computers do what computers are good at Data mining let humans do what they're good at Visual analytics

37 Data mining Visual Analytics Computer Graphics Human Computer Interaction Daniel A. Keim s perspective on visual analytics

38

39 Data Visualization Vis is hot & cool Vis is young What is vis research? What are we doing at HKUST?

40 What is vis research? Techniques/algorithms Applications Systems Evaluations Theory/models

41 Visualization Process Visualization Pipeline Engineering part Visual design part

42 Technique Papers Novel techniques or algorithms

43 Application/Design Study Papers Applying visualization and visual analytics techniques in an application area.

44 System Papers A blend of algorithms, technical requirements, user requirements, and design that solves a major problem.

45 Evaluation Papers Animations Traces

46 Theory/Model papers New interpretations of the foundational theory of visualization and visual analytics. How Information Visualization Novices Construct Visualizations Grammel et al. IEEE TVCG 2010

47 A Typical Visual Analytics Problem IEEE VAST Challenge 2009 An employee is leaking important information to the outside world; hypotheses about his identity and network need to be made or confirmed. There are three datasets: Badge and computer network traffic Social network (with a very small geospatial component) Video

48 VAST Challenge 2013

49

50 Role of Visualization in Big Data Analytics Scalability

51 Challenges for Visualization Research Scalability

52 Better Scalability - Better visualization - More people

53 Better Visualization Better overview & summarization Better data reduction Better visual encoding Better user interaction.

54 Top Ten Interaction Challenges in Extreme-Scale Visual Analytics P.C. Wong, H.W. Shen and C. Chen In Situ Interactive Analysis 2. User-Driven Data Reduction 3. Scalability and Multi-level Hierarchy 4. Representation of Evidence and Uncertainty 5. Heterogeneous Data Fusion

55 Top Ten Interaction Challenges in Extreme-Scale Visual Analytics 6. Data Summarization and Triage for Interactive Query 7. Analytics of Temporally Evolving Features 8. The Human Bottleneck 9. Design and Engineering Development 10. The Renaissance of Conventional Wisdom

56 How to Get More People? Lower the bar for visualizations Bring visualizations to masses

57 Lower the Bar for Visualizations Google Visualization API IBM s RAVE and Many Eyes D3.js Taobao s datav.org

58 D3.js

59 IBM Many Eyes

60 Taobao

61 百 度 Echarts

62

63

64 Data Visualization Vis is hot & cool Vis is young What is vis research? What are we doing at HKUST?

65

66 Medical Data Visualization TVCG 08 (Vis 08) TVCG 07 TVCG 09 1 (Vis 09) TVCG 2011 TVCG 09 2 (Vis 09)

67 High Dimensional and Relational Data TVCG 11 InfoVis 11 TVCG 09 (InfoVis 09) CGF 08 (EuroVis 08) TVCG 08 (InfoVis 08) CGF 09 (EuroVis 09)

68 Text Data Dynamic Tag Clouds OpinionSeer FacetAtlas IEEE CG&A 11 TVCG 10 (InfoVis 10) TVCG 10 (InfoVis 10) TextFlow TVCG 11 (InfoVis 11) TextWheel ACM TIST 12

69 Urban Informatics Trajectory Analysis VAST 11 Air pollution analysis TVCG 07 (Vis 07) Route Zooming TVCG 09 (Vis 09) Mobile Phone Data

70 Three Projects Twitter Data Visualization Trajectory Data Visualization Coursera Data Visualization

71 Social Flow

72 Whisper When and where is a story dispersed?

73 How information is spread? Query earthquake on Twitter

74 Information diffusion: When, where and how the ideas are spread? The temporal trend, social-spatial extent and community response to a topic of interest.

75 Challenges How can we represent the dynamic information diffusion processes for diffusion pattern detection and comparison? How can we deal with a huge amount of micro blog data in real time? How can we summarize the diffusion processes by considering temporal, spatial, topic and community information?

76 Visualization design Color hues: sentiments Color opacity: activeness of tweets / users Size: importance (number of followers) Shape: types (media / common users / retweeting activity)

77 Visualization design Diffusion series When mouse over on an active tweet, its diffusion series will be shown to illustrate ``when did who reweet whom.

78

79 Layout Topic Disc Diffusion Pathway User Group Assembling

80 Layout Topic Disc Diffusion Pathway User Group Assembling Sunflower Packing

81 Layout Topic Disc Diffusion Pathway User Group Assembling Diffusion Field Based on electronic fields Topic positive charged User group negative charged

82 Layout Topic Disc Diffusion Pathway User Group Assembling Flux line layout

83 Layout Topic Disc Diffusion Pathway User Group Assembling Users are grouped either by geo-locations / their shared interests Use voronoi icons to facilitate comparison Use sunflower packing for efficient layout in case of online monitoring Layout by longitude or evenly separate circle space

84 Layout Topic Disc Diffusion Pathway User Group Assembling Visual clutter reduction Reorder the user groups in match with the center active tweets to reduce crossings Rotate the topic disc to minimize the line lengths

85 Video

86 Stories (1) : tracing diffusion events Japan Earthquake

87 Temporal diffusion pattern (Medias)

88 Interview with Domain Experts Three experts: political scientist, design expert, communication phd Interview methodology: warm up; semi-structure interview with 7 questions; free discussions Results Discussions

89 A Complete VA System Design goals Underlying techniques (data mining; visualization, interaction, etc.) Case studies User study (questionnaire-based; task-based) Expert Interview

90 Visual Analytics of Trajectory Data Fix data errors Reconstruct continuous trajectories Route diversity analysis Data Error Data Uncertainty

91 Navigation System Microsoft T-drive [Yuan et al., 2010] a R 1 R 2 R 4 R 3 b 99

92 Encoding Scheme Time Outer circle Number of trajectories The height of the bars Duration Length of the arc Speed Color H. Liu et al. IEEE VAST

93 User Interactions Brushing by locations Brushing by time span 101

94 Traffic Monitoring 102

95 Route Suggestion 103

96 Route Suggestion 104

97 Embed Temporal Displays onto the Roads

98 Interchange patterns

99 Analysis of Clickstreams in the Coursera

100 628 Week 1-1 Week 1-2 Week 1-3 Week 1-4 Fig.1 Analysis on clickstream pattern-stacked graphs

101 Week 1-1 Week 1-2 Week 1-3 Week 1-4 Analysis on clickstream pattern-stacked graphs

102 Analysis on clickstream pattern - animation Fig.2

103 Fig.4 Analysis on clickstream pattern - animation 11 2

104 Analysis on user behavior 11 4

105 User classification Current Classification Users who watched only one video random users 28.1% Users who watched all videos persistent users 9.8% Users who watched more than one video and only those from the first seven videos impersistent users 44.5% Other types of users others 17.6% Comparisons between different types of users seek action most frequent and meaningful seek arc comparisons 11 5

106 Fig.10 Seek arc visual encoding scheme 11 6

107 Fig.11 Comparison of seek behavior on the first course video for different groups of users (500 users for each group) 11 7

108 Fig.12 Comparison among videos from different weeks for the user group who have watched all the 17 videos(persistent users) 11 8

109 *Specific Parts in Videos Corresponding to Histogram peaks Some patterns from peaks: 1) Skip the opening 11 9

110 *Specific Parts in Videos Corresponding to Histogram peaks Some patterns from peaks: 1) Skip the opening 12 0

111 *Specific Parts in Videos Corresponding to Histogram peaks Some patterns from peaks: 2) New term & Sensitive topic 12 1

112 *Specific Parts in Videos Corresponding to Histogram peaks Some patterns from peaks: 2) New term & Sensitive topic 12 2

113 *Specific Parts in Videos Corresponding to Histogram peaks Some patterns from peaks: 3) Findings vs. Conclusions 12 3

114 *Specific Parts in Videos Corresponding to Histogram peaks Some patterns from peaks: 3) Findings vs. Conclusions 12 4

115 Conclusions A primer on data visualization Examples to showcase HKUST vis projects Data visualization is hot, cool, and young

116

117 Q&A