Enabling the Big Data Commons through indexing of data and their interactions

Size: px

Start display at page:

Download "Enabling the Big Data Commons through indexing of data and their interactions"

Alicia Verity Gray
8 years ago
Views:

1 biomedical and healthcare Data Discovery Index Ecosystem Enabling the Big Data Commons through indexing of and their interactions 2 nd BD2K all-hands meeting Bethesda 11/12/15

2 Aims 1. Help users find accessible 2. Assist producers on how to publish for maximal discoverability 3. Build a prototype/platform to dock related products PubMed of Data = DataMed 2

3 Outline v The ecosystem w w Data, derived, meta Stakeholders v Nuts and Bolts w w Components: meta, search tool Plan and timelines v How to participate w w w Working groups Pilots Collaborations

4 What does it take to use big? v v v Find the (across various resources) Find the that operate on the Find the appropriate computational environment v v v Access the Access the (/systems) Access the computational environment

5 pre-curated meta preprocessed curated standards prep for ecosystem

6 Characterizing implies describing the that helped produce them preprocessed pre-curated curated meta standards prep for ecosystem

7 one woman s preprocessed are another woman s raw preprocessed pre-curated curated meta prep for ecosystem

8 hosting in a cloud, cluster, server preprocessed pre-curated curated meta standards prep for understanding which computational environment is best for the combination of and relevant is important (e.g., HPC, GPU) ecosystem

9 preprocessed selecting the right computational environment for the right type of is important pre-curated curated meta meta standards PHI curated PHI pre-curated PHI preprocessed curated Protected Health Information (PHI) hosting in a HIPAA cloud, cluster, server understanding the conditions of accessibility is also important ecosystem

10 pre-curated meta PHI pre-curated standards Repositories preprocessed curated + PHI curated PHI preprocessed curated big analytics depend on 1. merging from several different sources (e.g., reference bases, molecular repositories, clinical repositories), 2. proper, and the 3. proper computational environment ecosystem

11 pre-curated meta PHI pre-curated standards Repositories preprocessed curated + PHI curated PHI preprocessed curated prep for join prep for big projects use several types of digital objects and they are inter-related pre-curated meta preprocessed curated Centers Big Data projects prep for ecosystem

12 pre-curated meta PHI pre-curated standards Repositories results of analyses are too preprocessed prep for curated + join PHI curated prep for PHI preprocessed curated journal published results pre-curated meta selection post-processed (results) preprocessed pre-processed curated Centers, Big projects prep for ecosystem

13 new types of publications have emerged preprocessed pre-curated prep for curated meta standards + join PHI curated PHI pre-curated prep for PHI preprocessed curated Repositories journal published results pre-curated selection post-processed (results) preprocessed pre-processed prep for meta curated Centers, Big projects ecosystem

14 Stakeholders stakeholders have different responsibilities and interests funder owner producer big user curator manager host ecosystem

15 Stakeholders stakeholders have different abilities in indexing different types of funder owner producer big user curator manager host searching across different resources is time consuming because no one is an expert in all resources ecosystem

16 Data'Discovery'Index' searching across indices and repositories A' aggregator' B C A aggregator' existing indices can interoperate with the cross-aggregator index best indexers for are those who use it all the time, but they may not know as much about other resources '

17 find on Kawasaki disease Data'Discovery'Index' platform and portal A' aggregator' B C A aggregator' A, B, C: mapping of meta, standards, links to aggregators, passing of queries aggregators: various indices whose meta are or can be mapped into Commons meta ' digital objects

18 Meta Model A set of meta specifications, future-proofed for progressive extensions, to support intended capability of the Data Discovery Index prototype Created using

19 BioSharing: Content Standards and Databases Supported by the NIH grant 1U24 AI to the University of California, San Diego

20 Data Identifiers Define a set of best practices and operating procedures for identifiers that support the intended capability of the NIH BD2K Data Discovery Index (DDI) prototype - being designed by the biocaddie Core Development Team. Check document at biocaddie.org Attend breakout session on Identifiers

21 biocaddie Prototype Ingestion Indexing Repositories Meta Ingestion ElasticSearch Data Sources Online sets User Interface Funding Agencies Publishers Data producers Terminology server

22 Data Indexing Pipeline Data Source 1. Configuration file developed by curator 2. Extraction of meta/ from resource or set via ingestion module w Cache information for further processing 3. Process meta/ via a set of modules w e.g. ID conversion, keyword extraction, normalization 4. Mapping of meta/ to meta model(s) 5. Export to target endpoint(s) 6. Search via ElasticSearch APIs

23 User Interface Workflow Query Entry Entity Identification Expansion Query Execution biocaddie backend Advanced filters Terminology server ElasticSearch Presentation Facets Organize results Visualization

25 Core Development Roadmap DDI architecture Setup website for searching for sets Set up infrastructure for web portal Data identifier Implement Data identifier into the DDI Data indexing Set up indexing using meta from WG 3.0 Data ingestion Determine sets Decide on scalable /meta input routes Meta mapping September 2015 Version 0.1 Search function Implement the function for 3 repositories Feedback collection Github RFA for pilot on Harvester for DDI schema RFA announced Review, selection and award Wrap up of Y1 pilot projects Literature/set link: Advanced search Recommender System: Ranking results isee/delve: Innovative visualization PDB citation pipelines

26 Core Development Roadmap Dataset result display Sort sets Group meta Terminology server Import ontology Integrate to Scigraph API Integrate autocomplete feature to prototype Interface design New interface for prototype v 0.2 Global statistics Usability study UI Analysis Ranking algorithm Results from pliot project Search function Expand the function to 7 repositories Find similar sets Search history Architecture Code refactoring November 2015 Version 0.2 We are here

27 Core Development Roadmap Personalized search Share/save search results User account Link set to external resources PubMed Grants Search algorithm Boolean/advanced search Data repository search function Usability study User study Track user s action Ranking algorithm Refine search results based on user s selection Report from WG 8 on Ranking Data duplication problem Meta management February 2016 Version 0.5 June 2016 Version 1.0

28 v Working groups w Participate or follow v Prototype w Participation Using and providing feedback on the prototype search engine v Interoperate with the prototype w Link your favorite index Use or map to meta Collaborate on APIs v Recommend /repositories for inclusion w New working group 28

29 Pilot$ Newly awarded Meta Discover w w w Distributed discovery using gym: github, yaml and markdown Chris Mungall, Lawrence Berkeley National Laboratory Feasibility study of indexing clinical research using HL7 FHIR Guoqian Jiang, Mayo Clinic College of Medicine Meta discovery and integration to support repurposing of heterogeneous using the Openfurther platform Ram Gouripeddi and Julio Facelli, University of Utah

30 Working Groups

31 An Index of Data (Indices) Data'Discovery'Index' Organizing framework and portal for A' aggregator' B C A aggregator' dashed lines: mapping of meta, standards, links to aggregators aggregators: various indices whose meta are or can be mapped into Commons meta ' Data Digital objects

32 Acknowledgements 93 working group members 12 steering committee members 8 pilot application reviewers staff and trainees collaborators Supported by the NIH grant 1U24 AI to the University of California, San Diego

33 pre-curated PHI a mouse model for science pre-curated preprocessed meta curated + PHI curated join published pre-curated meta preprocessed PHI curated selection result preprocessed curated prep for

Searching biomedical data sets. Hua Xu, PhD The University of Texas Health Science Center at Houston

Searching biomedical data sets Hua Xu, PhD The University of Texas Health Science Center at Houston Motivations for biomedical data re-use Improve reproducibility Minimize duplicated efforts on creating