OVERVIEW OF DATA EXPLORATION TECHNIQUES. Stratos Idreos, Olga Papaemmanouil, Surajit Chaudhuri SIGMOD 2015, Melbourne

Transcription

1 OVERVIEW OF DATA EXPLORATION TECHNIQUES Stratos Idreos, Olga Papaemmanouil, Surajit Chaudhuri SIGMOD 2015, Melbourne

2 USER INTERACTION

3 express interests query/results recommendasons annotate collaborate visualize results User Interface Layer assisted query formulason

4 User Interface Layer

5 User Interface Layer Data Visualiza=on

6 User Interface Layer Data Visualiza=on Explora=on Interface

7 data visualizason visualiza=on tools User Interface Layer Data Visualiza=on Explora=on Interface visual op=miza=ons automa=c visualiza=on

9 (1,1) (M,1) Back in 1982 i th tuple goes here (1,i) window- based sophis=cated browser for rela=onal s browser for mulsple relasons/tuples TIMBER rich query language for icon- oriented s visual editor of text objects browser for geographical data TIMBER, VL 82

10 user- driven visual specs visualizasons Polaris back- end queries data cubes Polaris, INFOVIS 02

11 visual specificasons specificasons (abributes) user- driven visualizasons Polaris back- end queries data cubes back- end queries: data selecson, parsson into panes Polaris, INFOVIS 2002

12 visual specificasons transformasons (group by, sort) user- driven visualizasons Polaris back- end queries data cubes back- end queries: data transformasons (group, sort, aggregate within each pane) Polaris, INFOVIS 2002

13 visual specificasons mappings (shape, size, color) user- driven visualizasons Polaris back- end queries data cubes back- end queries: graphical transformasons (renter and visualize) Polaris, INFOVIS 2002

14 collaborasve explorason live annotasons Sky View explorason for sky objects/paberns AstroShelf, SIGMOD 12

15 Live Annota=ons collaborasve explorason subscripsons to interessng objects Sky View explorason for sky objects/paberns AstroShelf, SIGMOD 12

16 Live Annota=ons collaborasve explorason stream based nosficasons Sky View explorason for sky objects/paberns AstroShelf, SIGMOD 12

18 automasc visualizason request views User Interface Layer interessng? insigheul? Data Visualiza=on review views manual, repessve explorason for best visualizason(s)

19 auto- ranked visualizasons model good charts saved decks/ replay logs VizDeck search, select, promote, discard, save, share filter across charts, recommend, rank VizDeck, SIGMOD 12

20 automasc visualizasons user query Q 1 Q 2 uslity Q n high deviason from overall dataset aggregasons/ single- abribute group- by informa=ve queries % sales/ region sales over Sme visualizason engine See, PVL 13

22 resoluson reducson user query Sci query results Visualiza=on expensive, ineffecsve on big data sets Scalar, Big Data Vis 13

23 resoluson reducson user query Sci query results Visualiza=on user query Data Reduc=on reduced results Visualiza=on Sci modified query plans filter/aggregate/sample at given resoluson Scalar, Big Data Vis 13

24 approximate visualizasons user query Sampling reduced results approximate chart Visualiza=on SELECT X, AVG(Y) FROM R(X,Y) GROUP BY X original chart same group ordering Blais et al, PVL 15

25 approximate visualizasons user query Sampling reduced results approximate chart Visualiza=on SELECT X, AVG(Y) FROM R(X,Y) GROUP BY X clear ordering less samples Blais et al, PVL 15

26 approximate visualizasons user query Sampling reduced results approximate chart Visualiza=on SELECT X, AVG(Y) FROM R(X,Y) GROUP BY X correct order? sample more min # samples for correct order? Blais et al, PVL 15

27 approximate visualizasons user query Sampling reduced results approximate chart Visualiza=on SELECT X, AVG(Y) FROM R(X,Y) GROUP BY X #samples Group 1 Group 2 Group 3 Group 4 1 [60,90] [20,50] [10,40] [40,70] 20 [64,84] [30,48] [15,35] [45,65] 21 [66,84], I [30,48] [17,35] [46,64] 70 [66,84], I [40,47] [17,32], I [46,53] sampling phases/ confidence intervals Blais et al, PVL 15

28 visualizason management user query overlapping user queries query results Visualiza=on replicated db opera=ons memory opera=ons on big data Ermac, PVL 14

29 visualizason management user query query results Visualiza=on transforma=ons to pixel space visual op=miza=ons reduced visual rendering =me speciﬁca=ons DVMS logical visual plans è physical query plans Ermac, PVL 14

30 explorason interfaces automa=c explora=on User Interface Layer Data Visualiza=on Explora=on Interface assisted query formula=on novel query interfaces

32 manual vs automasc data explorason long, imprecise, labor- intensive process manual SQL query formulason query execuson result review predicate adjustment

33 manual vs automasc data explorason long, imprecise, labor- intensive process manual SQL query formulason query execuson result review predicate adjustment auto capture user interests op=mize query execu=on reduce user effort recommend data/queries

34 manual vs automasc data explorason long, imprecise, labor- intensive process manual SQL query formulason query execuson result review predicate adjustment auto capture user interests op=mize query execu=on reduce user effort recommend data/queries

35 explore by example relevant irrelevant User Model decision tree classifier Space Explora=on sample extrac=on Samples effecsveness vs efficiency sampling areas? sampling size? AIDE, SIGMOD 14/ VL 15

36 explore by example relevant areas to predict Abribute B Abribute A AIDE, SIGMOD 14/ VL 15

37 explore by example uniform sampling across domain x x x Abribute B x x x x x x Abribute A AIDE, SIGMOD 14/ VL 15

38 explore by example Abribute B x sampling around relevant objects predicted relevant area discover relevant area Abribute A AIDE, SIGMOD 14/ VL 15

39 explore by example sampling around boundaries Abribute B x refined predicted relevant areas Abribute A AIDE, SIGMOD 14/ VL 15

40 result recommendasons query results YMAL interes=ng queries addi=onal results YMAL, VLJ 13

41 result recommendasons query results YMAL interes=ng queries addi=onal results query extract query fasets expand adributes rank fasets top- k queries selecson predicates based on original query add abributes from table schema freq(result)/ freq() YMAL, VLJ 13

42 result recommendasons query results YMAL interes=ng queries addi=onal results query extract query fasets expand adributes rank fasets top- k queries selecson predicates based on original query add abributes from table schema freq(result)/ freq()!tle, year, genre of Scorsese movies!tle, year, genre, country of Scorsese movies + = many Scorsese movies are related to Italy YMAL, VLJ 13

44 keyword- based query suggessons SQL query (tedious) keywords (intuisve) relevant data relevant & irrelevant data keyword search relevant data how we can discover relevant queries? SQLSUGG, ICDE 11

45 keyword- based query suggessons keywords Template Matcher ranked templates SQL Query Generator suggested queries Sample Results/ Visualiza=on Template Repository database gray template on Stle/authors? template on Stle? SQLSUGG, ICDE 11

46 keyword- based query suggessons keywords Template Matcher ranked templates SQL Query Generator suggested queries Sample Results/ Visualiza=on Template Repository =tle year Paper Template 1 =tle year Paper id=p_id Template 2 Author template generason SQLSUGG, ICDE 11

47 keyword- based query suggessons keywords Template Matcher ranked templates SQL Query Generator suggested queries Sample Results/ Visualiza=on template relevance = f (en=ty relevance & importance) relevant template? Template Repository ensty relevance è keyword frequency in ensty ensty importanceè importance of data nodes SQLSUGG, ICDE 11

48 equi- join inference sample table A table B A 1 A 2 B 1 B 2 Cartesian product inference algorithm informa=ve tuple goal join predicate goal predicate: discover all posi=ves eliminate all nega=ves minimize user effort BonifaS et al, ET 14

49 equi- join inference sample table A table B A 1 A 2 B 1 B 2 Cartesian product inference algorithm informa=ve tuple goal join predicate (A1, B1) (A1, B2) candidate predicates (A1, B1) (A2, B1) (A1, B1) (A2, B2) (A1, B1) (A1, B2) (A2, B1) prune predicates with uninformasve tuples label tuple that prunes as many predicates as possible BonifaS et al, ET 14

50 graphical query specificason result visualizason answers non- answers DataPlay, PVL 13

51 graphical query specificason result visualizason answers non- answers pivot relason add, remove query constraints query /visualizason recommendasons seman=c query tuning by local syntac=c modifica=ons DataPlay, PVL 13

52 graphical query specificason result visualizason answers non- answers pivot relason add, remove results query correcsons search limited to local modifica=ons DataPlay, PVL 13

53 query recommendasons query results Charles queries selected query Charles, CIDR 13

54 query recommendasons query results Charles queries selected query different data parssons weight <5 >5 weight, height <5 >5 <20 <30 <5 >5 >20 >30 quality: simplicity, breadth, balance Charles, CIDR 13

55 query refinement condi=onal query select species from birds where color= {red: 80%, blue: 20%} Merlin ranked results by match probability sensi=vity of user predicates query refinements w/ quality improvement rank species 1 Bluebird 2 Blue Jay adr sensi=vity color 18.6 impact on ranking adr size quality score 83.3 legcolor 57.1 remaining adributes result quality if added in the query Merlin, ICDE 14

57 no- keyboard interfaces query context gesture recognison query intend query space search pabern query template Gesture, CIDR 13

58 no- keyboard interfaces novel database kernel touch input quick response touch recognison gesture recognison map touch to operators dbtouch, CIDR 13

59 InteracSve ExploraSon through Data Prefetching & Query ApproximaSon MIDDLEWARE TECHNIQUES

60 interacsve data explorason SQL query formulason query execuson result review predicate adjustment ad- hoc, non- op=mized, labor- intensive process interac=ve: small latency bounds on user wait =me

61 middleware opsmizasons query results query approxima=on online processing sample- based processing middleware prefetching specula=ve query execu=on result reuse structure- aware prefetching

62 sample- based processing query Samples sampling approximate results accuracy vs response Smes sample construcson & selecson error approximason

63 offline data synopses query Aqua transformed query samples histograms Synopses approximate results + confidence bounds join synopses: sample dissnguished joins congressional samples: biased sampling for group- by queries incremental maintenance: equi- depth & compressed histograms Aqua, SIGMOD 99

64 select avg(sessiontime) FROM table WHERE city= SF WITHIN 1 SEC online sample selecson online sampling selecson Results 190+/ (95% confidence) disk in- memory offline sampling on frequent columns sets parallel query execu=on on mul=ple samples across mul=ple machines samples across 1000s machines Blink, EuroSys 13

65 data impressions query & =me/error bounds approximate results Level 1 Level 2 Level 3 impressions during data loading adapsve sampling to explorason focus muls layer sampling and processing to meet user bounds SciBORG, CIDR 11

66 middleware opsmizasons query results query approxima=on online processing sample- based processing middleware prefetching specula=ve query execu=on result reuse structure- aware prefetching

67 speculasve query execuson Query Formula=on user wait =me Result Review =me Query Execu=on 1. predict follow- up queries 2. execute queries 3. cache results

68 speculasve query execuson Query Formula=on user wait =me Result Review =me Query Execu=on 1. predict follow- up queries 2. execute queries 3. cache results

69 speculasve query execuson Query Formula=on user wait =me Result Review =me Query Execu=on 1. predict follow- up queries 2. execute queries 3. cache results explora=on space reduc=on query enumera=on query ranking

70 cube explorason explora=on space reduc=on user query SELECT AVG (iops) FROM events WHERE month= m1 AND week= w1 GROUP BY zone month week hour itme zone center location rack DICE, ICDE 14

71 cube explorason explora=on space reduc=on user query SELECT AVG (iops) FROM events WHERE month= m1 AND week= w1 GROUP BY zone zone center location rack month week hour itme cube explora=on operators WHERE month= m1 WHERE month= m1 AND week= w1 AND hour= h1 WHERE month= m1 AND week= w2 DICE, ICDE 14 parent child sibling

72 cube explorason explora=on space reduc=on query enumera=on user query SELECT AVG (iops) FROM events WHERE month= m1 AND week= w1 GROUP BY zone month week hour itme specula=ve queries Q(month= m1 ) Q(month = m12 ) Q(hour = h1 ) Q(hour = h24 ) zone center location rack Q(week= w2 ) Q(week= w3 ) DICE, ICDE 14

73 cube explorason explora=on space reduc=on query enumera=on query ranking user query SELECT AVG (iops) FROM events WHERE month= m1 AND week= w1 GROUP BY zone month week hour itme specula=ve queries Q(month= m1 ) Q(month = m12 ) Q(hour = h1 ) Q(hour = h24 ) zone center location rack Q(week= w2 ) Q(week= w3 ) DICE, ICDE 14

74 cube explorason Query Formula=on user wait =me, t Result Review =me Specula=ve Execu=on Query Execu=on DICE, ICDE 14 74

75 cube explorason Query Formula=on user wait =me, t Result Review =me Specula=ve Execu=on QUERY Probability Exec Time Q Q Q Q Q Query Execu=on maximize query probability total speculason Sme < t DICE, ICDE 14 75

76 result reuse prefetching window prefetching window Query 1 Query 2 Query 3 Execu=on Execu=on Execu=on =me idensfy (likely) overlapping results cache them reduce query execuson Sme (user wait Sme)

77 semansc windows user- defined window properses overlapping results/windows SW 4 SW 1 SW 3 SW 2 window prefetching which order? 2D explorason space Kalinin et al, SIGMOD 14

78 semansc windows user- defined window properses overlapping results/windows SW 4 SW 1 SW 3 SW 2 uslity- based result ranking & result prefetching 2D explorason space Kalinin et al, SIGMOD 14

79 semansc windows extend & prefetch SW 1 SW 2 online performance vs query compleson Sme adjust prefetching size to output progress Kalinin et al, SIGMOD 14

80 query diversified results k representa=ve tuples with max total pairwise distance data diversificason

81 query diversified results k representa=ve tuples with max total pairwise distance data diversificason Query Output Max Diversified Set Search Diversified Output k= 3 T 1 T 2 T 3 T 4 T 5 d(t 1, T 3 ) d (T 2, T 3 ) random tuple d(t 4, T 3 ) d(t 5, T 3 )

82 query diversified results k representa=ve tuples with max total pairwise distance data diversificason Query Output Max Diversified Set Search Diversified Output k= 3 T 1 T 2 T 3 T 4 T 5 d(t 1, T 3 ) d (T 2, T 3 ) random tuple d(t 4, T 3 ) d(t 5, T 3 ) T 1 T 2 T 3 T 4 T 5 d (T 2, T 1 )+ d(t 2,T 3 ) d (T 4, T 1 )+d(t 4, T 3 ) d (T 5, T 1 )+d(t 5, T 3 )

83 query diversified results k representa=ve tuples with max total pairwise distance data diversificason Query Output Max Diversified Set Search Diversified Output k= 3 T 1 T 2 T 3 T 4 T 5 d(t 1, T 3 ) d (T 2, T 3 ) random tuple d(t 4, T 3 ) d(t 5, T 3 ) T 1 T 2 T 3 T 4 T 5 d (T 2, T 1 )+ d(t 2,T 1 ) d (T 4, T 1 )+d(t 4, T 3 ) d (T 5, T 1 )+d(t 5, T 3 ) T 1 T 2 T 3 T 4 T 5

84 interacsve data diversificason w w w w w Q 1 w Q 2 w w w w Q w 3 w w overlapping diversified results long Time- To- Insight cache diversified results and use most promising regression model predicts max diversificason of a set DivIDE, SSM 14

85 interacsve data diversificason query Cached Diversified Results reusable results query results divide search space reusable diversified results new query results model based output selec=on diversified results search space pruning through regression model best/first fit search for max total diversificason among cached and new results DivIDE, SSM 14

86 structure- aware prefetching prefetching for interacsve spasal query sequences model structures of past spasal queries in graph idensfy guiding structure in past two queries : iterasve pruning cache the predicted next locason SCOUT, VL 12