Deep Carbon Observatory Data Science Interim Report, Dec

Transcription

1 Deep Carbon Observatory Data Science Interim Report, Dec Progress on the main issue, problem or subject and why is it important. The Deep Carbon Observatory (DCO) Data Science (DCO-DS) team has spent the ~ first year of this phase implementing key data science infrastructure, developing data management plans and policies, and working with and further developing relations and assessing data science requirements from the four Science Communities (Deep Energy, Deep Life, Extreme Physics and Chemistry, and Reservoirs and Fluxes), DCO Engagement and DCO Secretariat. The result has been substantial progress on all six key elements of DCO-DS identified in the original proposal, as well as several additional sub-tasks, and refinement of priorities. We have in place the software infrastructure for ~ 95% of the proposed initial data science platform that coexists with, and enhances, key community activities. As such we have implemented the initial version of the Deep Carbon Virtual Observatory (DCVO); a collaborative scalable education and research environment for searching, accessing, integrating, and analyzing distributed observational, experimental, and model databases. We also have made progress in facilitating the definition of each of the following: A Virtual Mineral Laboratory, The Global Census of Deep Life, The Global Census of Deep Fluids, The Global Volcano Observatory, The Global State of High Pressure and Temperature Carbon (and related) Materials, and A Global Inventory of Diamonds with Inclusions. A key leveraging step has been the identification and endorsement (by the Mineralogical Society of America) of an Earth Materials Data Infrastructure (EMDI) spanning all DCO Science Communities. The genesis of EMDI arose from the data infrastructure elements as part of DCO- DS; in the Deep Carbon Virtual Observatory, Leveraging Community activities, Science Network, and Visualization and Discovery and comprise ~90% of the DCO-DS effort (personnel and funds) in this project period. The DCO-DS platform (Fig. 1 and 3) is facilitating collaboration, data and object registration, materials repository and much more. 1

2 - Figure 1. Schematic for the DCO Data Science Platform. The Web-based portal (built on Drupal; top of figure) is underpinned by three key data framework components. These are the DCO repository (left center) based on the Comprehensive Knowledge Archive Network (CKAN), the DCO research network (center) based on VIVO, and the identifier infrastructure based on the Global Handle System (GHS; lower). To the upper right and center right are visual renderings of the DCO science network (based on the initial 63 registrants on the DCO portal, i.e. ~ March 2013). Key Elements of the Data Science Effort Given DCO s intensive data and computational needs, each of the activities embedded in the Science Directorates and instrument development initiatives of the Deep Carbon Observatory have adopted or adapted data science and data management solutions to fulfill both decadal strategic objectives and their day-to-day tasks. As noted earlier, the Data Science effort for DCO will continually assess in detail the data science and data management needs in each DCO activity and for the DCO as a whole, by using a combination of informatics methods; use case development, requirements analysis, inventories and interviews. Detailed records of progress and activities can be found at: 2

3 Priorities for DCO-DS have been established by the Data Science Advisory Committee on an intermittent basis as input to the DCO-DS PI and Project/ Technical leads to ensure a balanced allocation of DCO-DS resources and to advance DCO decadal goals. We report on the six key elements of a DCO-DS platform (with actual percentage of the data science personnel effort for the project period in parenthesis): i) the Deep Carbon Virtual Observatory (40%), ii) Leverage and enhancement of the existing community data resources (25%), iii) the Deep Carbon Observatory Science Network (a virtual organization) (20%), iv) Visualization and scientific exploration (5%), v) Data as a first-class object, and vi) Data Science education and working activities (v-vi comprise the remaining 10%). i. The Deep Carbon Virtual Observatory Figure 2. Conceptual Information Model for the DCO research/data ecosystem. Using our use-case driven methodology we have formalized the information modems applicable to the DCO (See Fig. 2). This information model has guided our development of the underlying schema (ontology) for the data infrastructure and allowed us to leverage and extend existing schema from VIVO, CKAN, etc. All schemas are open source and being fed back to the respective communities (as well as software integration/ modification). 3

4 - Figure 3. Data Science Platform elements and relations. The elements are presented schematically in Fig. 1 and their relations are indicated here. Of note is that the primary support for the DCO infrastructure comes from the VIVO framework ( and that CKAN and GHS are further removed (i.e. not explicitly visible) ii. Leverage and Enhancement of Community Data Resources Generating, assembling and analyzing the libraries of new and complex data created by the DCO requires management of the inherent complexity to allow integration of information and knowledge across multiple scales spanning traditional Figure 4. Screen shot of live DCO data portal for Noble Gas Isotope dataset. Shown are the DCO-ID, QR code for easy recognition and (lower) dataset metadata fields. disciplinary boundaries. Curation of these data is also a critical element of such assembly. 4

5 Figure 5. Screen shot of dataset detail for collection in Fig. 4. This effort has been an exemplar for DCO Data Science Data Deposition and has benefited from very strong support from Deep Energy leadership (Cole/ Sheets; OSU). To date, we ve worked very closely with the Deep Energy Community, particularly for the Noble Gas Isotope dataset (see Fig. 3). We ve also further developed strong relationships via Boundary Activities with EarthChem, PetDB (geochemical data) and SESAR (sample registry) at Lamont Doherty Earth Observatory/ Columbia University ( for the Reservoirs and Fluxes Community, MetPetDB at RPI ( genomic data with taxonomic counts, sequences, and Visualization and Analysis of Microbial Population Structures [VAMPS] at the Marine Biology Laboratory ( for Deep Life. We have started and will build on integrating and further populating the thermochemical databases, such as the Library of Experimental Phase Relations [LEPR] with Mark Ghiorso as identified in the EPC Community proposal. ( Another undertaking for which DCO-DS dedicated resources toward, is the volcano gas emission database. While this began with the Global Volcanism Program at Smithsonian, it has not shifted to LDEO and we remain in close contact with these groups and ready to provide links to the DCO data infrastructure. In preparation for these efforts, as well as for larger DCO sample activities, RPI (DCO-DS) became a full member of the International Geo Sample Network registry (approve at AGU Dec. 2013) and is licensed to issue IGSN identifiers for DCO samples. 5

6 - We plan to actively continued these community data activities in the current and next phases. A welcome opportunity is provided by the DCO summer school for which the conveners (Jones, Baross) have indicated a strong desire for DCO datasets to be made available for school participants, as well as data science material embedded in the curriculum. We have committed to participating in the summer school as part of this effort. iii. The DCO Science Network. Figure 6. Northeast American continental regional map of DCO science network participants (~Nov. 2013). This map is illustrative of what one view of the network will be once more participants are entered (either registered for logins, or their data added). As part of the progression toward a global community of carbon scientists, DCO has the opportunity to form a virtual science network. The Data Science Team worked extensively with the Engagement Team in developing and refining key functions for the DCO web portal support collaboration. In doing so, there was a need to characterize roles and relationships within the DCO Science Network and to identify key liaisons within each project team. All such 6

7 functions are now implemented. All that is required is the population of the content, a task that extends well beyond the purview of the Data Science Team. Special effort will be required over the next 2-3 months in order to substantially increase the formal number of participants in the science network (meaning they exist and are part of the network now but not visible through the portal). That effort involves the DCO secretariat, Engagement and the larger DCO community. iv. Visualization and Scientific Exploration The Data Science Team has begun its contributions to visualization and data exploration in the first year as planned by working directly with several DCO scientists (specifically Hazen, Downs and Sverjensky) to both determine suitable visual approaches as well as finding ways of lowering the cost of visualization generation and allow greater integration in the scientific process. Figure 7. Klee diagram: A 2D matrix of all 72 x 72 essential mineral-forming elements, in which each matrix element is the percent of minerals with essential element E1 on the axis that also has element E2 on the Y axis (from Hazen, Downs, Fox et al. in preparation). Fig. 7 is an example of an early Klee diagram produced in the exploration of essential mineral forming elements. The diagram was produced by graduate student Yu Chen (RPI) working with Hazen and a graduate student of Downs. The results were well received and 7

8 - indicate strong potential for increasing such visualization efforts across more of DCO (e.g. exploratory work with Sogin on accessing Census of Deep Life records via VAMPS and generating Linked Data (RDF) has also been encouraging). v. Data as a first class science object An early contribution of the Data Infrastructure Team was the establishment of the identification infrastructure for DCO scientific data objects (DCO-ID based on the Global Handle System, Fig. 1). These identifiers facilitate the long-term management, discovery and exploitation of the range of artifacts beyond data. These artifacts include reports, papers, visualizations, tables, presentations, data products, interpretative analyses, etc. produced by DCO researchers. Integrated with this naming infrastructure is a rich metadata management system (using VIVO; see Fig. 1) enabling data integration, federation and the construction of higher-level applications and visualizations in the DCVO; see Fig. 1. Web services allow query (whether by users or applications) using the identifier for key metadata, access means, derivative products, provenance records, related artifacts of all types, etc. distinguishing citations attributable to DCO ( - e.g / CC = Noble Gas Dataset, or directly ). vi. Data Science Education and Data Management Training To date, attention to Data Science in DCO has increased over the first year. Marshall Ma will attend the early career scientist workshop in 2014 and present and encourage data science. Data, and data science has explicitly been included in the plans for the 2014 DCO Summer School. The Data Science Team has presented elements of Data Management and Data Science at the Deep Energy workshop in Manchester UK early in The Team also developed the DCO data policy guidelines to be synergistic with the publication and sharing guidelines. These are to be online at: Further, detailed data management and planning documentation was developed early in the first year 8

9 this document can be found at: and will migrated to the main DCO web site in Metrics/Goals Overall Goal and Metrics provided by Sloan Foundation to RPI: To develop methods and tools to handle DCO data integral with efforts of other institutions, programs and disciplines as shown within 18 months by posted protocols, packages of code, and agreements with other organizations such as NSF 1. To capture data and encode and store modeling and simulation results from the DCO to have at least 6 early examples with 18 months We will complete this goal by May To innovate with visualization of large data sets, as shown by at least 6 examples in publications or used widely in DCO presentations We will complete this goal by May To provide underlying services for the DCO that facilitates routine collaboration as shown by use by most of the core DCO researchers within 18 months We have completed the goal of providing the services and collaboration functions. The DCO Data Science Team does not have primary (or secondary) responsibility for adding DCO researchers (via registered logins) to the web site. As such DCO relies upon DCO Engagement, DCO Secretariat and the DCO Communities themselves to promote the use of the DCO Science Network. The Data Science Team will remain very responsive to look-and-feel aspects of the collaboration functions. 2. Additional Results from partnering with the DCO Secretariat, Science Directorates, DCO Engagement and Broader Community Reporting: In the first year, a Community Reporting function for DCO was identified. Due to the flexible architecture and interfaces provided both to DCO content (based on information model in Fig. 2), the Data Science Team were easily able to put in place the technical infrastructure for such reports. An example is shown in Fig. 8. A fully working 9

10 - capability will appear in 2014 as interface and query requirements are finalized. Figure 8. Early rendering of the Community Reporting function for DCO. This requirement was suggested/ identified during the first year of DCO. While the entire technical infrastructure is in place, the formatting of the reports, as well as specific queries to generate the reports are still being defined. DCO Secretariat, Engagement and Communities will have responsibility for adding content to the DCO repositories. The Data Science Team will continue to facilitate this activity. Boundary Activities: In the first year, the Data Science Team has initiated 7 major boundary activities as outlined below (See Appendix A for specific timing and level of activities compared to proposed ones). Boundary activities in DCO-Data Science are tasks to establish data infrastructure interfaces (data, metadata and services) that allow all community resources to be a part of the DCVO as well as allowing DCO data to flow into these locations such that it is still 10

11 known and attributed to DCO and discoverable. Details on each of these including status (and as additional activities are added) can be found at: DS/WorkingGroups/BoundaryActivities. 1 Reservoirs and Fluxes Community (DCO-RF) 1.1 Global Volcanism Program (GVP) Gas Emission Extension 1.2 Integrated Earth Data Applications (IEDA) and International Geo Samle Number (IGSN) 1.3 Diamonds and Mantle Geodynamics of Carbon (DMGC) and DiamondDB 1.4 Bibliography for volcanic gases and fluxes 2 Deep Energy Community (DCO-DE) 2.1 Igor's Noble Gas Dataset 3 Extreme Physics and Chemistry Community (DCO-EPC) 3.1 EPC Computer Cluster 3.2 Populate a New Data Source for Properties 4 Deep Life Community (DCO-DL) 4.1 Visualization and Analysis of Microbial Population Structures (VAMPS) 5 Geoscience Data Journal (GDJ) 5.1 Application to be an approved data center of Geoscience Data Journal 6 AGU Index List 6.1 Utilization of AGU Index List in the DCO Data Portal 7 DCO Summer School 2014 (spans all Communities) 7.1 Datasets and Data Science for Day One of Summer School. Bibliographic Infrastructure: 11

12 - Figure 9. Object registration workflow for DCO. This workflow allows for content to be hosted in other locations and have other identifiers, as well as be updated. The Data Science Data Infrastructure Team worked with the DCO Engagement Team to implement a DCO-wide Bibliographic Infrastructure (VIVO) that incorporates the digital object /name resolution infrastructure, extensible metadata model, efficient ingest workflows, and semantic discovery capabilities to track and manage an ever-growing list of DCO publications and their associated digital objects (data, with provenance, see Fig. 9). File and Information Sharing: The Data Science Data Infrastructure Team has implemented a Web-based file- and information-sharing approach using a combination of the Drupal file repository and the CKAN (see Fig. 1) repository. The relationships are denoted in Fig. 3. These resources are available to all DCO researchers (with login) and are being used for managing information, resources, and digital objects associated with research needs in data science and data management as well as communications and management functions. New semantic discovery tools will be available in 2014 for researchers to more effectively explore the holdings. 12

13 Coordination: The DCO-DS has worked extensively with the Engagement Team to enhance communications among DCO researchers and ensure that their scientific data and other artifacts are accessible and public under appropriate data-access policies. 4. Who has worked on this project in first year (Nov Nov. 2013). PI Peter Fox (Professor, project lead, architecture and design, resource management, executive committee member) Jim Hendler (Professor, architecture and design, resource management) John Erickson (Operations Director, operations, project management) Marshall Ma (Post-doc, data management, boundary activity lead, visualization) Patrick West (Senior Software Engineer, technical lead, primary developer) Yanning Chen (PhD candidate, software developer, and visualization technical lead) Han Wang (PhD candidate, software developer, and visualization technical lead) Katie Dunn (Technical services and metadata librarian, preliminary, effort will increase in 2014 for first 3 months) Dan Molik (student systems administrator, Nov 2012-Jan 2013) Anusha Akkiraju (MS student, summer 2013) Chengcong Du (MS student, summer 2013, fall 2013) Mengyu Yin (MS student, summer 2013) 13

14 - Appendix A. DCO Data Science primary activities and initial schedule. During the first period of DCO-DS activities (their approaches are discussed above) are Data Science needs analysis; Data Science outreach and education; Community engagement; Science network; Visualization; File and information sharing; Community database; Bibliographic infrastructure; Visual materials repository; Coordination; and Boundary activities: GVP, VAMPS, EarthChem, DE#1, EPC#1 (the latter two are to-be-determined boundary activities from Deep Energy and Extreme Physics and Chemistry Communities) and their approximate schedules are presented in Table J. These activities were phased according to our present understanding of DCO data needs and will need to adapt to changing requirements. Table J. As performed activity schedule for the first three-year period of the project. Activities and any sub-tasks are introduced in Section 2. =major effort, x=minor effort. Project start Nov Green = activity successful, Orange = activity behind or delayed, Red = activity cancelled or not achieved. Activity / Phase Year 0.5 Year 1 Year 1.5 Year 2 Year 2.5 Year 3 Data Science needs analysis. x Data Science outreach and education x x Community engagement/ coordination x x Science network x Visualization products File and information sharing. x Community database x x Bibliographic infrastructure x Visual materials repository x x Boundary Activities: GVP/emission VAMPS x EarthChem x x x EPC#1 x DE#1 x x Workshop 14

15 15