DataBridge Arcot Rajasekar" The University of North Carolina at Chapel Hill "

Transcription

1 DataBridge Arcot Rajasekar" The University of North Carolina at Chapel Hill "

2 Data Bridge: A Social Network for Long Tail Science Data" Outline of the Talk" Motivation" Design" Implementation Status" Examples" Future" " Data Bridge" 2"

3 Big Data" Well-known problem" Characteristics:" Volume" Exponential Increase in Size & Count" Velocity" Speed of Generation & Consumption" Variety" Disparate Types of data" Veracity" Integrity & Fidelity" Value" Worth" cognizan(.weber.org infographics.socialnama.com Data Bridge" 3"

4 Three Kinds of Big Data (1)" Archetypal Big Data" Science Projects LHC, LSST, SCEC, OOI " Business/Industry Genomics, Finance, Pharma " Government NASA, NOAA, NCDC " " Volume High large datasets" Velocity High but predictable" Variety Low Standardized, Metadata, Curated" Veracity High Fidelity and Credible" Value High funded" Findability High Known sites and discovery mechanisms" Availability High Published API" Light & Visible Data " infographics.socialnama.com wired.com 4"

5 Three Kinds of Big Data (2)" Crowd-Sourced Big Data" Social Media Facebook, Twitter " Recommenders Yelp, Angie s List, Groupon" Web Commerce Amazon, Ebay, Orbitz, enews" Volume High small data" Velocity High and non predictable" Variety High But well managed" Veracity Mixed Low to High" Value Ephemeral can be None to High" Findability High Known and Advertised" dreams(me.com Availability Immediate Interest Web pages and Apps" Nova-like Data " Data Bridge" 5"

6 Three Kinds of Big Data (3)" Long-tail Big Data" Science Projects small teams and organizations" Personal Hobbies, Amateur/Citizen Science/Arts" Government Internal and unpublished" Volume High small data sets" Velocity Low" Variety High Too many, Idiosyncratic" Veracity Non Credible until proven" Value unknown" Findability None Hidden and not advertised" Availability None In local, disks and tapes" Dark Data " teradata.com Images.frompo.com 6"

7 Our Interest: Long tail of Science Data" Large number of data generators" Highly distributed" NSF in 2011" 11,150 awards" Median size $126K" Primarily single PI" Data individually not petascale but large in aggregate" Of possible Value " Dark Data" Unpublished" Used once and Forgotten" Sunset Data" Even NSF retention expects at the least only 3 years after project" Expired Data " curated but disposed" Social media data" com2733amandaathens.blogspot.com 7"

8 Problems: Long-tail of Science Data" First Mile Problem" How to make it available?" Where do I upload?" Who is in charge?" How do I get credit?" Can I control access?" How do I pool with other like-minded researchers? Community services?" How much is long-term?" Who pays for it?" Last Mile Problem" How to make it findable?" What is needed to make it more visible? Metadata?" Are there other methods to make my data findable?" My data has specific ways & characteristics" How do I expose " them as finding " aids?" How can I find similar " data?" Solving the long-tail problem will also help " other two Big Data problems " blog.enrichconsul(ng.com Data Bridge" 8"

9 Dark Data from The Long Tail of Science " Long tail data amounts to small data sets produced by numerous investigators." Dark Data Exemplars:" From Brahe to Mendel, discovery has come from relatively small data sets" Much long tail data is dark data, data not easily found by potential users (Bryan Heidorn)" Long tail data sets lack structural advantages of classic Big Data, such as professional curation and homogeneous formats and well-documented data formats and populated & well-formed metadata schema." Improving availability and findability will help " solve the problems with this type of big data " and make it more mainstream." " Expose the hidden nuggets" astrosolar.com 9"

10 Data Bridge: A Social Network for Long Tail Science Data" Outline of the Talk" Motivation" Design" Implementation Status" Examples" Future" " Data Bridge" 10"

11 Data Bridge: A solution" We tackle mainly the last mile problem" But show an avenue for solving the first mile problem" Main Aim: Improve Findability" Solution: Empower Data!!" Empower data to find its own community" Community of likeness" Similarity in multiple dimensions" Look at data from diverse angles" Find " Relationships strong links " Friendships weaker links" Assist scientists in discovering " interesting data sets by automatically " forming communities of data" " Data Bridge" 11"

12 Data Bridge Strategy: Automatic Community Detection " Doc Watson albums at Amazon.com" Clustered by Yasiv.com" Clusters represent " related items" Clusters are connected by some internal " metrics " Data Bridge" 12"

13 Data Bridge: Design" Construct multi-dimensional social networks for data. " Three challenges:" Evaluate multiple types of " metrics on data" Domain-specific, genre-specific, " project-specific" Use Socio-metric Network Algorithms" Similar to but for data" Find relevance" Slices of similarity" Explore Relationships between " Data, Users, Resources, Methods, Workflows " Use Relevance Algorithms" Create communities" Use Clustering Algorithms" Provide an extensible & big data framework" Democratize the process" Data Bridge" 13"

14 Data Bridge Infrastructure" Accommodate for multiple, extensible number of " SNA, RA and CA algorithms" Provision an easy way to add new algorithms" Crowd sourcing" MyVector (add your own way of defining metrics)" Provision an easy way to connect data to algorithms" Multiple ways of finding similarities" Multiple ways of providing search criteria" MyBridge (add your own way of finding relevance)" Provision an easy way to form communities" Multiple ways of categorizing data" MyCommunity (add your own " domain-specific clustering)" Make it a distributed system " Grow and Shrink as needed" Make it easy for third-party setups" Federation of Data Bridge" " Data Bridge" 14"

15 Data Bridge Architecture" 15"

16 Message-oriented Architecture" Loosely linked processes: Messages make the connections" Scenario: " User A has a novel, signature detection algorithm for gene sequences" User A wraps algorithm with API provided by DataBridge" Subscribes to a message for gene sequence data " User B publishes a new gene sequence into DataBridge" A new message is created informing the new addition" User A s algorithm detects, computes relevance to signature " Publishes a new relevance message for this signature " User C has a relevance algorithm that catches A s message" Uses it to add B s sequence to a new data community" Other signature detectors may also look at B s gene sequence" Publish their own relevance, if applicable" User A can also look at " older gene sequences and " find relevance" Image from support.oyala.com Data Bridge" 16"

17 Data Bridge: A Social Network for Long Tail Science Data" Outline of the Talk" Motivation" Design" Implementation Status" Examples" Future" " Data Bridge" 17"

18 Modules: Current Status " RabbitMQ Messaging system" Ingest Engine" Relevance Engine" Network Engine" Ingestion GUI" DataVerseNetwork access" Meta database" MongoDB" Network Database" Neo4J" Viz Display " Data Bridge" 18"

19 Messages" Message Listener Originator Ingest Metadata Ingest Engine Any Data Provider (DVN) Metadata Available Relevance Engine Metadatabase Create Similarity Matrix Relevance Engine Any Ingest Engine Similarity Matrix Available Network Engine Any Relevance Engine Insert Similarity Matric Ingest Engine User/App Run SNA Algorithm Network Engine Network Engine or user/app SNA Data Available Network Engine Network Database Create Visualiza(on Data Network Database Network Visualiza(on Show Visualiza(on Visualizer User (WebApp) Data Bridge" 19"

20 Example Message Schema" Name Header Insert.Metadata.Java.URI.MetadataDB System headers type subtype User provided headers classname namespace inputuri Example Value databridge ingestmetadata Example Value org.renci.databridgecontrib.ingest.mockingest system_test /projects/databridge/metadata.xml W3.org Data Bridge" 20"

21 Data Bridge: A Social Network for Long Tail Science Data" Outline of the Talk" Motivation" Design" Implementation Status" Examples" Future" " Data Bridge" 21"

22 Screenshot: Finding similarities" Select Network Data Filter Connec(vity by similarity value Data Bridge" 22"

23 Screenshot: Weight of similarity" Similarity measure: "

24 Screenshot: Highlights of similarities" Link to the data 24"

25 Screenshot: Data Access" Data Bridge" " 25"

26 Screenshot: Simple Ingest GUI" Data Bridge" " 26"

27 Data Bridge: A Social Network for Long Tail Science Data" Outline of the Talk" Motivation" Design" Implementation Status" Examples" Future" " Data Bridge" 27"

28 Next Steps" Basic Framework implemented " Applied to a few thousands of datasets" Work to do, advanced features" Documentation" Scaling tests" More types data/metadata to be tested" Ready for new algorithms" Ready for more data" Ready for larger usage" Investigate multiple" similarity measures" Usage, Methods as " relevance" Data Bridge" 28"

29 Players" Howard Lander" Justin Zhan" Merce Crosas" Gary King" Jon Crabtree" Tom Carsey" Sharlini Shankaran" Arcot Rajasekar" Data Bridge" 29"

30 Arcot Rajasekar" The University of North Carolina at Chapel Hill " Conclusion" DataBridge" Motivation" Design" Implementation Status" Examples" Future"