Reagan Moore, PI Mary Whitton, Project Manager. National Science Foundation Cooperative Agreement: OCI

Transcription

1 Reagan Moore, PI Mary Whitton, Project Manager National Science Foundation Cooperative Agreement: OCI

2 DFC to Support Hydrologic Modeling Jon Goodall and Bakinam Essawy University of Virginia DFC October 2014 NSF Review Slide 2

3 Motivation Reproducibility Automate data gathering, transformation, and ingestion steps required for modeling. Create reusable, reproducible, and sharable data analyses and visualizations from large data collections (e.g., model outputs). Use virtual machine images so that results can be reproduced. DFC October 2014 NSF Review 3

4 Why Data Grids? Problem: Hydrology datasets are Distributed Multiple owners/curators Heterogeneous in structure and semantics Vision: Co locate data transformations with data sets Server side data processing to support science Reduce burden on scientists to complete repetitive data transformation steps Improve performance by reducing network traffic

5 Why Cloud Computing? We are using Amazon Web Services (AWS) for our computing resources Specifically, we are using Elastic Computing 2 (EC2) instances from AWS for our computing resource. This allows us to easy scale up or scale down our computing needs based on project demands. We can easily image instances and reproduce machines later, if needed.

6 Using Cloud Computing with irods An EC2 instance was launched, and the irods server installed on it Server connected to the icat located at RENCI. irods server is for the University of Virginia (UVA resource). Another EC2 instance launched at UVA: Acts as the client, as only the icommands were installed DFC October 2014 NSF Review Slide 6

7 Launched instances on the AWS Using Putty to connect instance on AWS Instance on the AWS DFC October 2014 NSF Review Slide 7

8 Using FilleZilla client to transfer files in/out the EC2 instance on AWS Files copied to the AWS instance

9 Integrated Rule Oriented Data Management System (irods) DFC October 2014 NSF Review Slide 9 Source:

10 How the AWS Interacts with irods irods Collection (/hydrology/hom e/bakinam)./iget or./iput Command EC2 Instance Client IP ( )./iget or./iput Command SSH (rules) EC2 Instance for irods Server IP ( ) SSH (workflow) Personal Computer IP ( ) 1. Rule is on client but executed on the irods server. 2. Rules are located on the bin directory for the client. 3. Workflow (scripts on the irods Server (/home/jlg7h/irods/server/bin/cmd) directory. DFC October 2014 NSF Review Slide 10

11 Last Demo Data Processing VIC model input (pre processing stage) Source : Billah et al. (2014) DFC October 2014 NSF Review Slide 11

12 Demo Objectives Use the Terra Populus Integrated Data on Population and Environment data along with hydrology model outputs to better understand drought impacts. Automate the creation of visualizations from the TerraPop data and model output files using the DFC and irods Workflow Structured Objects (WSO). Share the visualization output and WSO used to create the visualization through the SEAD HydroShare Project Space. DFC October 2014 NSF Review Slide 12

13 The VIC Model VIC = Variable Infiltration Capacity; A regional scale land surface hydrology model VIC developed at UWashington and Princeton; applied worldwide Spatial resolution: 1/8 degree grid cell Three layers of soil: top layer (Layer 0, 0 10cm) mid layer (Layer 1, 10 30cm) lower layer (Layer 2, cm) Source : Gao et al. (2009) DFC October 2014 NSF Review Slide 13

14 VIC Output data set Variable Infiltration Capacity (VIC) Macro scale Hydrologic Model VIC data set show/45 VIC Output data set on the irods server DFC October 2014 NSF Review Slide 14 Example for a flux file. Fluxes_x_y: x = latitude, y = longitude flux files contain information about moisture and energy fluxes for each time step for the three layers of soil (Top, Middle, and Deep).

15 Executing the WSO to Visualize the VIC Model Output 1. VIC model output datasets and scripts are located on the irods server VIC Output dataset VIC processing scripts run_psp_vic_soilmoisture.scr spatiotempdatabase.py vic_calc_mnth_mc.py vic_monthly_soilmoisture.py vic_soil_moisture.py 2. From irods client, workflow file and parameter file are put into an irods collection irods Client irods Server vicsoilmoisture1.mss vicsoilmoisture.mpf irods collection vicsoilmoisture.rundir 4. *.run file is accessed by irods Client, workflow is executed on irods server using specific parameter file. Execution of workflow generates a new directory and output is stored there. vicsoilmoisture.run 3. irods generates a *.run file

16 County Level Population Data From TerraPop From TerraPop we obtained: 1. A shapefile for all the counties in the US with a unique identifier called GEOID. 2. A.csv file that includes the GEOID, counties name, total population, and the cropland area. DFC October 2014 NSF Review Slide 16

17 Automating TerraPop Data Retrieval using the DFC Grid From Terra Populus collection on irods dfcmain grid to the irods client local machine From irods client local machine irods collection on the hydrology grid irods collection Dfcmain grid iput irods collection Hydrology grid Dfcmain/home/TerraPop boundaries_us_slad_2010.shp data_us_slad_2010.csv iget /hydrology/home/bakinam/terrapopdata boundaries_us_slad_2010.shp data_us_slad_2010.csv Retrieving TerraPop Data from irods collection on hydrology grid to irods resource executable directory by using iget command irods Server Bin directory iget Extract assembled in TerraPop data access system and sent to TerraPop collection on irods dfcmain grid /var/lib/irods/irods/server/bin/ boundaries_us_slad_2010.shp data_us_slad_2010.csv

18 Impact on Science The ability for scientific communities to collaborate around repeatable, reusable, and sharable data visualization workflows that operate on large data collections in DFC The ability to include datasets from other communities as resources within the data visualizations (e.g., TerraPop) The ability to easily share visualizations with metadata through SEAD DFC October 2014 NSF Review Slide 18

19 Who is DFC? DFC October 2014 NSF Review Slide 19

20 DataNet Federation Consortium Drexel Engineering/ MRC TDLC/ Scripps / DICE iplant Collaborative RENCI/ Arizona State U Virginia UNC CH/ Odum/ SILS/ DICE DFC October 2014 NSF Review Slide 20

21 National Science Foundation Cooperative Agreement: OCI

22 1 Rule is on client but executed on the irods server. 2 Rules are located on the bin directory for the client. 3 Workflow (scripts on the irods Server (/home/jlg7h/irods/server/bin/cmd) directory.

23 Science Observatory Network (SciON) Name Institution DFC October 2014 NSF Review Slide 1

24 SciON Goals High level Goals SciON: Science Observatory Network Seismic, Marine and Environmental Science Network SciON and DFC federated to provide persistence for sensor data Immediate Goals: Enable ingestion of sensor data into DFC Ingest streaming real time seismic data from NSF USArray Transportable Array Ingest streaming real time atmospheric pressure data from NSF USArray Transportable Array Ingest streaming real time seismic data from the Central and Eastern US Network Continue to provide data collected by the OOI as available DFC October 2014 NSF Review Slide 2

25 Importance of Real Time Sensor Data Provides real time view of sensing environment Provides ability for early warning systems for earthquakes, tsunamis, wild fires, severe weather, ocean waves Provides oceanographic data in near real time for oceanographic analysis and modeling DFC October 2014 NSF Review Slide 3

26 Antelope Environmental Commercial Software Data Collection and Analysis Monitoring Software Antelope is an integrated collection of programs for data collection and seismic data analysis, and typically runs at the central processing site. It has been in development for over a decade and is deployed around the world. Near real time Processing The Antelope Real Time System is built around a large, flexible, non volatile ring buffer. Data acquisition modules communicate with data loggers, and leave data on the ring buffer. The ring buffer protocol provides a convenient method for directly importing data from other sites, as well as exporting data. Real time processing typically occurs on the ring buffer: programs take input from the ring buffer and write results to the ring buffer. For instance, the detector reads data from the ring buffer and writes detections to the orb. The grid associator reads the detections and quickly provides preliminary event locations. This architecture facilitates running multiple detectors or associators, and other refinements. DFC October 2014 NSF Review Slide 4

27 More on Antelope Open Architecture Antelope runs in a UNIX environment on Linux X86, Linux X Scale, and Macintosh OS X. Antelope has an open architecture, with extensive documentation of internal interfaces. There is already support for many common dataloggers and sensors, but other dataloggers, sensors, and novel devices may be integrated by the end user. Integrated Database A relational database underlies Antelope real time processing. Waveforms, detections, events, and other data are saved from the ring buffer into the database. The analyst reviews data in the database, and there are database versions of programs corresponding to the real time processing software. Development Environment Antelope is a development environment. There is extensive documentation, allowing development of specialized software for site specific applications. Languages supported include C, Fortran, Perl, TCL/Tk, and Python. The Antelope Toolbox for MATLAB is also included for locations that already have MATLAB installed. The Antelope Users' Group has collected source code generated by the Antelope User Community DFC October 2014 NSF Review Slide 5

28 Antelope Usage United States: USArray Plate Boundary Observatory Alaska Earthquake Center University Nevada Reno US Air Force Australia Austria Canada Chile Italy Saudi Arabia Oman Korea Malaysia Taiwan Antarctica DFC October 2014 NSF Review Slide 6

29 USArray Data Flow DFC October 2014 NSF Review Slide 7

30 Science Impact Seamless integration of data from different science disciplines

31 Years 4 and 5 Make interface to Antelope more robust Access Antelope waveform data from different types of sensors Access other types of streaming data available in Antelope instances Creation and deposition of Archival Sensor Packages into NetCDF resources (eg THREDDS Server) Promote applications using asynchronous federation subscription, notification, playback Provide oceanographic data as available DFC October 2014 NSF Review Slide 9

33 DFC to Support Social Science Data Preservation and Access Jonathan Crabtree Odum Institute University of North Carolina at Chapel Hill DFC October 2014 NSF Review Slide 1

34 Odum s Multidisciplinary Mission Promote diverse social science research Culture of multidisciplinary collaboration Methodological training programs Leveraging data science Focus on data reuse and research transparency History of data stewardship Dataverse Network partner Founding member of Data-PASS DFC October 2014 NSF Review Slide 2

35 Social Science Archives in Collaboration for Preservation Strategic partnership agreements Coordinated Operations Joint best practices Shared federated catalog Shared tools & technologies DFC October 2014 NSF Review Slide 3

36 Model for Sustainability Strategies Diverse funding sources Strategic partnerships Multiple business models within partnership Diverse holdings that appeal to wide array of disciplines Data-transfer agreements that ensure sustainability DFC October 2014 NSF Review Slide 4

37 Odum s Overall Demo Goals Design curation workflow integration Connect research environment with archive Connect archive with national architecture Open source focused As pluggable as possible DFC October 2014 NSF Review Slide 5

38 Bringing great tools together irods Dataverse Modeshape Databook Architecture Apache Service Mix irods Rule Integration Indexing Engine DFC October 2014 NSF Review Slide 6

39 Dataverse Storage Abstraction Current production environment tied to UNIX based file system In this prototype we used Modeshape Expands the storage options for Dataverse Abstracts the storage layer Allows standardized interface to irods Allows future use of Rules Based Policy Management and ties to Dataverse Data Tags DFC October 2014 NSF Review Slide 7

40 Leveraging DFC Infrastructure Databook infrastructure Rules base policy management Distributed preservation environment Secure distributed storage based on policy rules DFC October 2014 NSF Review Slide 8

41 Odum Demo DFC October 2014 NSF Review Slide 9

42 Advantages for Social Scientists Simplifies archive connection to active research environment Adds possibility for rules based policy management reducing data sharing burdens Enhances secure data sharing possibilities Diversifies Dataverse preservation options Allows curators in archive to assist researchers with data management within individual workflows Gives researchers access to diverse data from many disciplines DFC October 2014 NSF Review Slide 10

43 Impact on Science This DFC infrastructure will act as a catalyst for innovative combinations of digital data. Seamless discovery of data from diverse disciplines could be a fundamental shift in how researchers gather and use digital data. USE CASE Hydrologist uses Dataverse multidisciplinary search. Topic: impacts of drought Result: finds researchers down the watershed engaged in similar issues (e.g. Coastal estuary damage and its effect on fisheries, changes in oceanographic currents and their effect on weather patterns DFC October 2014 NSF Review Slide 11

47 DFC to Support The Science of Learning Andrea A. Chiba Science Director Temporal Dynamics of Learning Center DFC October 2014 NSF Review Slide 1

48 DFC October 2014 NSF Review Slide 2

49 Our Goal: Understand the role of time and timing in learning From the neuronal and millisecond scale to the brain and multi yearscale DFC October 2014 NSF Review Slide 3

50 Temporal Dynamics of Learning Center: Overview DFC October 2014 NSF Review Slide 4 4

51 The Network of Networks UC San Diego Brown University Carnegie Mellon University Imperial College Rutgers University San Diego State University The Salk Institute UC Berkeley U. at Buffalo University of Colorado U. Pennsylvania U. Pittsburgh SensoriMotor Network Mercado Social Interaction Network Jernigan Serpell Schultz Cognitive Science Cognitive Psychology Computational Neuroscience Computer Science Developmental Psychology Education Learning Theory Linguistics Machine Learning U. Queensland U. of Victoria U. of Washington Vanderbilt University Virginia State University Interacting Memory Systems Perceptual Expertise Network Mathematics Neuropsychology Neuroscience Physics Robotics DFC October 2014 NSF Review Slide 5

52 Actual Collaborations SIN PEN SMN IMSN DFC October 2014 NSF Review Slide 6

53 TDLC requires support for geographically distributed collaborations that share large datasets across tasks and species, coordinate joint analyses of data, share novel stimulus sets, and support computational modeling, while ensuring IRB, HIPAA, and IACUC data restrictions are maintained DFC October 2014 NSF Review Slide 7

54 Distributed Challenges: DFC October 2014 NSF Review Slide 8

55 DFC October 2014 NSF Review Slide 9

56 Bridging Time and Space with irods irods enables geographically distributed collaborative science, bringing laboratories together through a venue for large scale, realtime data sharing Sharing data in accordance with human and animal subjects restrictions requires the implementation of rules for sharing DFC October 2014 NSF Review Slide 10

57 Team Science DFC October 2014 NSF Review Slide 11

58 Distributed Challenges: DFC October 2014 NSF Review Slide 12

59 Video experimental data uploaded using idrop web Video data downloaded for use in Australia DFC October 2014 NSF Review Slide 13

60 Local neural data bulk uploaded using idrop web Processed Collections of Data Received and uploaded from Boston: Collections accessed from Australia for use in models: DFC October 2014 NSF Review Slide 14

61 Remote IRODS Clients Local IRODS Clients MMAP d Hi-Speed Direct I/O IRODS application Virtualized I/O DataDirect SFA Grid Enabled Storage KVM Driver DDN RAID Stack Linux Kernel Technological Sustainability SSD SAS SATA DFC October 2014 NSF Review Slide 15

62 Impact on Science The ability for scientific communities to collaborate on large data collections; even repurposing data for robotic models The ability to continue to collaborate with other science of learning centers and share and publish data collections as desired The ability to adhere to institutional review board standards for sharing and protection of data while also using a virtual platform DFC October 2014 NSF Review Slide 16

63 Future Directions Establish standards for different collections from different scientific domains. Enable data analysis on the grid, using each scientific domain s tools. Create easy Interface for the Rules Engine (for control of each dataset for IRB and IACUC restrictions). Develop portals for sharing data training sets for student education (Interface with MOOCS). DFC October 2014 NSF Review Slide 17

67 iplant: Data to discovery CI for data intensive life sciences Nirav Merchant iplant/university of Arizona DFC October 2014 NSF Review Slide 1

68 The iplant Collaborative: Vision Enable life science researchers and educators to use and extend cyberinfrastructure Slide 2

69 iplant Architectural Motivation We strive to be the CI Lego blocks Danish 'leg godt' 'play well Also translates as 'I put together' in Latin If a solution is not available you can craft your own using iplant CI components DFC October 2014 NSF Review Slide 3

70 iplant Products DFC October 2014 NSF Review Slide 4

71 The Community Overall Registered users upwards of 20,000 Each product has approx unique users per week Each iplant product has unique data access and use patter Central nature of data make the iplant data store essential component along with Auth service that powers all iplant platforms DFC October 2014 NSF Review Slide 5

72 Data Challenges for iplant Common for each user to have 100 GB 2 TB data Team science and virtual organizations (VO) are very prevalent, often global and highly distributed Consortiums have TB data to share Data movement, sharing, leveraging compute resources are a necessity. Connect data with compute Compute resources are provided by multiple partners Support for modern ways to share and collaborate with data driven teams (web, api, cloud etc.) Establish iplant Data Commons, improve discoverability DFC October 2014 NSF Review Slide 6

73 Milestones and Accomplishments, Year 3 Test grid of the iplant has been integrated with DFC Integration of the iplant data grid with the DFC data grid in progress The policy requirements were assembled. Several micro services used for integrity and end to end encryption Developed by Technology Group at request from iplant Tuning and Optimization of Metadata Catalog for improving performance Performed by Technology Group at request from iplant Generalization of the Discovery Environment in process Exposing iplant s HPC support to DFC Community in progress Generalized add in for popular desktop based analysis applications (NIH imagej and Tassel) to access irods were developed and will be made available for the DFC Community DFC October 2014 NSF Review Slide 7

74 iplant: Life Sciences/Biology Demo Community iplant Data Providers NGS Researcher (Next Gen. Seq.) idrop icommands Image Analysis (NIH imagej) TASSEL (Genetics) Jargon Plugins 30GB+ files 5GB + files iplant Discovery Environment Metadata driven iplant Data Store (irods) Large knowledge repositories (Genome Browsers) UCSC and ENSEMBL UK Requests data by range Creates interactive visualization for web and desktop systems Ecologist (Environmental Data) Bounding Box + date range irods Fuse iplant Atmosphere DFC October 2014 NSF Review Slide 8 NEW NASA MERRA A/S (1970 to date) Runs Map Reduce Returns NetCDF for further analysis in Atmosphere

75 Years forward 1. iplant Federation with other sites (in US and UK) 2. HIVE/metadata integration for our Data Commons effort 3. Metric and understanding of how our data store is being utilized to improve data management capabilities and planning for growth 4. Scalability for irods fuse, data distribution via content delivery systems (read and write) 5. Data Carpentry especially for Next Gen Sequencing and imaging users 6. Enhance support for spatial data, LIDAR and use of container technology (Docker) for processing data directly on resource servers DFC October 2014 NSF Review Slide 9

76 Impact of DFC Ability to provide tools and capabilities to break the 2GB (web protocol imposed) barriers for data transfer Allow users to manage their data with ease, connecting compute, data and facilitating analysis at scales required by their domain Wide range of interfaces developed Avenue to share best practices, metadata management New collaborations e.g. NASA MERRA A/S DFC October 2014 NSF Review Slide 10

77

78 Milestones for Years 4 5 DFC October 2014 NSF Review Slide 12

81 Demo Takeaways Name Here Affiliation DFC October 2014 NSF Review Slide 1

82 Demos: Takeaways These are not point solutions; this is about cyberinfrastructure, applications can tie together A base architecture has been developed that allows pluggable features, and the demonstrated features are examples of plug ins. The DFC architecture and plug ins support preservation, sharing, and discovery by merit of storing your data on the DFC grid, and those capabilities are available via any client. Plug ins are compact, and allow easy extension to accommodate use cases in different domains. DFC October 2014 NSF Review Slide 1

83 Goals Index/data sci. Representation/ontologies Preservation/ discovery Challenges continue as the DFC grows Diversity of formats, domains, terminologies Impediments discoverability, usability, etc. Milestones Impact Interoperable metadata/data infrastructure Data that is sustainable, discoverable, searchable, accessible, usable; can be repurposed Support, even expedite science Support better science long term DFC October 2014 NSF Review Slide 2 Format Registry HIVE Databook

84 Motivations Preserve data in neutral formats Maintain metadata about significant properties, provenance Share databy federation Control data by policies Maximize data value by facilitating curation and discovery DFC October 2014 NSF Review Slide 3