iplant + irods: Enabling data driven collaborations Nirav Merchant iplant Collaborative/Univ. of Arizona nirav@ .arizona.edu VAMP 2012 Utrecht

Transcription

1 iplant + irods: Enabling data driven collaborations Nirav Merchant iplant Collaborative/Univ. of Arizona nirav@ .arizona.edu VAMP 2012 Utrecht

2 Topic Coverage About iplant 4 th Paradigm Technology challenges for life sciences iplant Data Store (ids) Challenges of Sharing Data iplant Atmosphere (cloud) Future: Identity+Group+Network with Openflow

3 What is iplant The iplant Cyberinfrastructure Collaborative is building a comprehensive informatics infrastructure for plant biology. Funded by the National Science Foundation (NSF) 2008 (and continuing till 2018) This rapidly evolving infrastructure is sometimes very visible to users (researchers), and sometimes absolutely transparent to them (projects powered by iplant components).

4 The iplant Collabora/ve Cyberinfrastructure Philosophy We have designed iplant to be consistent with the pillars of CIF21* ü High Performance Compu?ng ü Data and Data Analysis ü Virtual Organiza?on ü Learning and Workforce

5 Science Paradigms 1. Thousand years ago: science was empirical describing natural phenomena, observations 2. Last few hundred years: theoretical branch using models, generalizations 3. Last few decades: a computational branch simulating complex phenomena 4. Today: data exploration (escience) unify theory, experiment, and simulation Based on the transcript of a talk given by the late Jim Gray to the National Research Council Computer Science and Telecommunication Board in Mountain View, CA, on January 11,

6 The Fourth Paradigm: Data-Intensive Scientific Discovery Increasingly, scientific breakthroughs will be powered by advanced computing capabilities that help researchers manipulate and explore massive datasets. The speed at which any given scientific discipline advances will depend on how well its researchers collaborate with one another, and with technologists, in areas of escience such as databases, workflow management, visualization, and cloud computing technologies. 6

7 The Discovery Lifecycle The Fourth Paradigm: Data-Intensive Scientific Discovery 7

8 Big Data (yes we have it!)

9 Data- intensive biology necessitates biologists become comfortable with new technology (rather quickly)

10 10

11 One key goal in our infrastructure, training and outreach is to minimize the emphasis on technology and return the focus to biology MaG Meselson & Ultracentrifuge, $500, Sharp, Sambrook, Sugden Gel Electrophoresis Chamber, $250

12 Ways for users to access iplant Atmosphere: cloud compu?ng plaqorm Data Store: secure, cloud- based data storage Discovery Environment: a web portal to many integrated applica?ons (combine data + compute) DNA Subway: genome annota?on, DNA bar- coding (and more) for science educators Founda/on API: For programmers embedding iplant infrastructure capabili?es (Auth, IO, Apps, Jobs, Dir. etc) Command line: for expert access (thru TeraGrid/XSEDE)

13 The iplant Cyberinfrastructure End Users Teragrid XSEDE Computa?onal Users

14 The iplant Discovery Environment A rich web client Consistent interface to bioinforma?cs tools Portal for users who won t want to interact with lower level infrastructure An integrated, extensible system of applica?ons and services Addi?onal intelligence above low level APIs Provenance, Collabora?on, etc.

15 Scalable Computa/on for High- Throughput Inquiry 90,000 Compute Cores Up to 1TB shared memory TACC Lonestar TACC Ranger Growing to ~500,000 cores by end of 2012 PSC Blacklight TACC Corral EBI Web Services

16 iplant Layered Services and Access" End Users iplant Data Store" Scalable" Reliable" Redundant" High-Throughput" Computational Users

17 Powered by iplant The iplant CI is designed as infrastructure. This means it is a platform upon which other projects can build. Use of the iplant infrastructure can take one of several forms: Authentication (~IdM/P, Shib, CAS etc) Storage Computation Application Hosting Web Services Scalability 17

18 Powered by iplant Other major projects are beginning to adopt the iplant CI as their underlying infrastructure (some completely, some in limited ways): BioExtract (computation) CiPRES (authentication, computation) Gates Integrated Breeding Platform (hosting, development, authentication) Galaxy (storage, for now) CoGE (authentication, data store, hosting) Many more (check 18

19 CIPRES Portal Federation

20 irods Developed by Data Intensive Cyber Environments (DICE) Directed by Reagan Moore Developed SRB, the Storage Resource Broker at SDSC, the San Diego Supercomputer Center Most of the group migrated to UNC Chapel Hill in (The group is bi-coastal: DICE-UNC, DICE-UCSD) Released irods, the integrated Rule-Oriented Data System, in 2009 Primary development funding from NSF (and other agencies)

21 irods Data grid middleware Data management infrastructure A framework for procedural implementation of data management policy (policy-driven data management)

22 Resource + Catalogue Server(s)

23 iplant Data Store" Free Your Data" Different Users, " Different Access Needs: " One Data Store"

24 iplant Data Store (ids)" WebDAV DE API i-commands idrop

25 The iplant Data Store Fast data transfers via parallel, file transfer Move large (>2 GB) files with ease Mul?ple, consistent access modes iplant API iplant web apps Desktop mount (FUSE/DAV) Java applet (idrop) Command line (icommands) Tickets and tokens Fine- grained ACL permissions Sharing made simple Access and a storage alloca/on is automa/c with every iplant account

26 Some Challenges Allowing 3 rd party apps to users data Used irods rules for ACL handling E.g. Bisque Image Analysis (updates their web app of data deposition in bisque_data iplant Data Store) SSO allowed jumping between 3 rd party apps (and internal) *Users want to give access to files, directories for download & upload! (anonymous/non iplant/apps) Integrated tickets (tokens) Foundation API (REST access) *Users want fine grain access to permission Restrict access from certain domain (*.arizona.edu) for jobs running on other compute grids (UWisc, OSG) Enhanced tickets to allow host, group, file count, size based control * Funded by NSF OCI : (Klingenstien, Koranda, Merchant)

27 iticket! Niravs-MacBook-Air:$ iticket iticket>ls id: string: X23MQI8I5H70O0e ticket type: write obj type: collection owner name: nirav owner zone: iplant uses count: 0 uses limit: 0 write file count: 0 write file limit: 10 write byte count: 0 write byte limit: 0 expire time: none collection name: /iplant/home/nirav/ticket-incoming No host restrictions No user restrictions No group restrictions

28 irods+shib Building on the ASPiS solution from King's College, which allows web-based applications to be Shib enabled This solution leverages the Apache SP, and manages user accounts based on provided Shib attributes and entitlements Allows customization of behavior by providing a standard set of irods rules and micro services The ASPiS solution is being updated for inclusion in the Java Jargon library, and the out-of-the-box idrop web interface. ASPiS option will run by setting a configuration option that runs a Shibboleth-aware servlet filter. Testing is currently underway on the integration of the ASPiS approach and this integration should be available by end of September Funded by NSF OCI : (Klingenstien, Koranda, Merchant)

29 Customized cloud platform for computing on your terms!

30 Atmosphere: motivation Standalone GUI-based applications are frequently required for analysis GUI apps not easily to transform into web apps Need to handle complex software dependencies (e.g specific bioperl version and R modules) Users needing full control of their software stack (occasional sudo access) All computation does not complete in a 24 hour queue (HPC limitations!) Need to share desktop/applications for collaborative analysis (remote collaborators) Availability of Next Gen map-reduce based algorithms (currently we have limited support)

31 Challenges of existing cloud platforms Amazon Web Services (AWS) Flexible and scalable High level of expertise required for configurations Fairly challenging for biologists to master all steps Limited lifecycle management (cost, time mgmt ) Lack easy desktop integration Lack easy tools for large data transfer

32 Steps to get started!

33 What is Atmosphere? Self-service cloud infrastructure Designed to make underlying cloud infrastructure easy to use by novice user Built on open source Eucalyptus Fully integrated into iplant authentication and storage and HPC capabilities Enables users to build custom images/ appliances and share with community Cross-platform desktop access to GUI applications in the cloud (using VNC) Provide easy web based access to resources

34 The iplant Collabora/ve Project Atmosphere : Custom Cloud Compu?ng API- compa?ble implementa?on of Amazon EC2/S3 interfaces Virtualize the execu?on environment for applica?ons and services Get Up to 12 core / 48 GB instance Access to Cloud Storage + EBS Big data and the desktop are co- local again Bring your data to Atmosphere VM for interac?ve access and analysis Send it back to the DE for transac?onal analysis >60 hosted applica?ons in Atmosphere today, including users from USDA, Forest Service, database providers, etc. (30 more for postdocs and grad students for training classes)

35 Atmosphere: Collaboration iplant Data Store

36 Lifecycle

37 How to Connect

38 Different Ways to Log in to VMs

39 Atmosphere: Launch a new VM

40 Atmosphere: Access a running VM

41 Atmosphere: Log in via shell

42 OpenFlow+OpenStack(Quantum) Allow custom network topology for specific user defined groups Placement of irods resource server and instances Floodlight handles the Openstack integration and openflow switch management Use RESTful interface from Grouper Placement of instance within our distributed infrastructure is easier (prevent vlan pain) Apply group policy, security, data access at network level (not in VM which is easy to override as sudo/ root)

43 Challenges Collaboration as ad-hoc groups/teams is becoming increasingly prevalent Defined groups need to persist across ALL our infrastructure Combining the capabilities of cloud, network virtualization and iplant Data store (with understanding of groups!) Our apps are web + command line (need a happy medium for emerging auth systems!)

44 Will Computers Crash Genomics? Science Vol. 331 Feb 2011

45 Acknowledgement All amazing iplant staff, students, collaborators and community members! Too many to name!!