irods for Big Data Management in Research Driven Organizations Charles Schmitt CTO & Director of Informatics RENCI

Transcription

1 irods for Big Data Management in Research Driven Organizations Charles Schmitt CTO & Director of Informatics RENCI

2 Acknowledgements Presented work funded in part by grants from NIH, NSF, NARA, DHS, as well as funding from UNC Teams involved include: DICE team at UNC and UCSD Networking team at RENCI and Duke Data sciences team at RENCI UNC Dept of Genetics, Research Computing, Lineberger Comprehensive Cancer Center, NC Tracs Institute, Center for Bioinformatics, Institute for Pharmacogenetics and Personalized Treatment UNC HealthCare Multiple members of the irods community 2

3 RENCI Researches, Develops, and Deploys Cyberinfrastructure Tools Networks Joint Venture between UNC, Duke, NCSU, and State of North Carolina Virtual Organizations evaluate Visualization Projects Collaborators Data Science of Cyberinfrastructure improve High Performance Computing Funding Scholarship Innovation Engagement Software Analytics 3

4 RENCI Key Initiatives E1: Storm Surge Modeling E2: NSF SSI (HydroShare) E3: PIRE Environmental/Coastal Sciences Biomedical and Health Sciences H1: CTSA H2: Sequencing H3: Secure Med. Workspace HPC H4: Decision Virtual Organizations Visualization Support C1: S2I2 C2: REACH NC Data Science of Cyberinfrastructure Networks C5: CIBER (NARA) C6: ORCA/BEN (NSF GENI) C3: E-iRODS C4. DataNet Software Analytics Tools E-iRODS GeoViz SRW 4

5 Use Case: Informatics for Next-Generation Genomics ~ Whole and Exomic Sequences generated ~10,000 Sequences stored RENCI Next Gen Sequencing Sequencing Informatics Computational Workflows, High Performance Computing, Distributed Data, Security Informatics & Cyberinfrastructure R&D Clinical Practice Identifying genomic variants relevant to clinical care Exploring ethical/legal issues around reporting genomic findings Clinical Research Determining relationships between genomic variants and disease Basic Research Finding new ways to understand the relationship between genes and disease/behavior In collaboration with UNC Research Computing, UNC Dept of Medical Genetics, Lineberger Comprehensive Cancer Center, Institute for Pharmacogenetics and Personalized Treatment, UNC High Throughput Sequencing Core, UNC Center for Bioinformatics 5

6 Managing Research Data: Genomics Sequencers Tape Archive Initial Pipeline, QC Alignment Pipeline, QC Data/Information Flow - managed by: 1) Multiple Custom Workflow Management Systems Archives (NIH, Library?), Replication Variant Detection Analysis: Phasing, Imputation, IBD, Phenotype Correlation R&D New Methods Clinical Decision Support & Presentation 2) Multiple Custom Laboratory Information Management Systems (LIMS) 3) E-iRODS Variant Database, Hadoop Clinical Validation Clinical Review Clinical Binning External Data Feeds (RefSeq, OMIM, PolyPhen, ) 6

7 The research data ecosystem: challenges UNC STORAGE (Tape, Drives) RENCI STORAGE (Tape, Drives) Genomics Storage Lab Machines Open Science Grid Teragrid External Partner Resources UNC HPC RENCI HPC RENCI Hadoop Genomics HPC Genomics Hadoop IT Machines Clouds Data management challenges: Analysts Wild West Automated Processes Controlled Developers Tracking data and metadata Data movement and migration Enforcing policies, compliance, security Encouraging managed automation Cost, disk and IT time Failures Data Providers External Partners IT Staff While not disrupting access to data Students Compliance While goals, processes, users, and software change

8 Big data and new stressors More data munging, more tools and processes involved, more hardware, more people (esp. IT and CS people) More security and compliance concerns More QC and QA concerns Data too big for review + more people/process=more mistakes More infrastructure breakages: storage systems, software tools Time slows down and mistakes are more costly Moving data is a planned IT event, analysis take days to months 8

9 What s needed? A multitude of technologies that play well together LIMS, analysis workflow engines, HPC queues, RDBMS, archival and library systems, web reporting/submission sites, Middleware that: Ties together the technologies Automates data-related chores Virtualizes the IT data infrastructure Securely manages the data at scale Works within a dynamically changing research environment Presentation title goes here 9

10 Integrated Rules Oriented Data System (irods) Proven in production use: NASA, NOAA, National Archives, Max Planck Society, Broad Institute, Wellcome Trust Sanger Institute, Lineberger Comprehensive Cancer Center, Bejing Genome Institute, Dow Chemical, Merck, International Neuroinformatics Coordinating Facilities, Proven at scale: iplant - 10k users; French National Institute for Nuclear Physics and Plasma Physics - 6 PB; Australian Research Collaboration Service storage resources; NASA Center for Climate Simulations million attributes; Cinegrid sites across Japan-US-Europe Solid foundation: SRB: initial product (developed by DICE Group, owned by General Atomics) in 1997 irods: rewrite of SRB by DICE Group in 2006; currently on version 3.3 Enterprise irods: mission critical distribution co-developed by RENCI and DICE in 2012 Support: Community of developers from groups worldwide Independent groups offering consulting and support and development irods Consortium offering formal support, training, involvement, and development help 10

11 irods- high level view Research Community Research Group A Research Group N - Unified logical interface to data and metadata resources (single namespace) - -based management of access - -driven management of data (replication, deletion, ) Institution A repository Archivals Institution B repository PI data sets Community data collections Data Services

12 irods Key Features Unified and consistent name space for digital objects Centralized metadata system Tagging, queries, used for process and security controls Manages digital objects stored in a variety of systems NFS, HDFS, S3, DDN WOS, HPSS, Instantiated via web service call, REST call, SQL query, Hadoop job, Multiple clients and APIs enforcing distributed rule engine 12

13 Principals of driven data management Relational model from late 60s/early 70s Foundation for SQL and RDBMS systems model Foundation for policy based data management systems Presentation title goes here 13

14 -based Data Management Purpose Defines Collection Defines Property Defines Controls Procedure SubType Updates Persistent State Information Ex: - QC check run - File integrity validated Periodic Assessment Criteria Source: Reagan Moore

15 -based Data Management - Collection Purpose Defines Collection Defines Digital Object Attribute Updates Property Defines Controls Procedure SubType Updates Persistent State Information Periodic Assessment Criteria Source: Reagan Moore

16 -based Data Management Collection Properties Purpose Defines Collection Defines Digital Object Attribute Integrity Updates Authenticity Access control Completeness Feature Feature Property Defines Controls Procedure Feature SubType Periodic Assessment Criteria Updates Persistent State Information Correctness Feature Consensus Consistency Source: Reagan Moore

17 -based Data Management Collection Policies Purpose Defines Collection Integrity Defines Replication Checksum Quota Data Type Digital Object Updates Attribute Authenticity Access control Completeness Feature Feature Property Defines Controls Procedure Feature SubType Periodic Assessment Criteria Updates Persistent State Information Correctness Feature Consensus Consistency Source: Reagan Moore

18 -based Data Management Collection Procedures Purpose Defines Collection Integrity Defines Replication Checksum Quota Data Type Digital Object Updates Attribute Authenticity Access control Completeness Feature Feature Property Defines Controls Procedure Feature SubType Periodic Assessment Criteria Workflow Chains Updates Persistent State Information GetUserACL SetDataType Correctness Feature Function SetQuota Consensus DataObjRepl Source: Reagan Moore Consistency Operation SysChksumDataObj

19 -based Data Management Persistent State Purpose Defines Collection DATA_ID DATA_REPL_NUM DATA_CHECKSUM Integrity Defines Replication Checksum Quota Data Type Digital Object Updates Attribute Authenticity Access control Completeness Feature Feature Property Defines Controls Procedure Feature SubType Periodic Assessment Criteria Workflow Chains Updates Persistent State Information GetUserACL SetDataType Correctness Feature Function SetQuota Consensus DataObjRepl Source: Reagan Moore Consistency Operation SysChksumDataObj

20 -based Data Management Enforcement Purpose Defines Collection DATA_ID DATA_REPL_NUM DATA_CHECKSUM Integrity Defines Replication Checksum Quot a Data Type Digital Object Updates Attribute Authenticity Access control Completeness Feature Correctness Feature Property Defines Controls Procedure Feature Feature Enforcement Point SubType Periodic Assessment Criteria Workflow Chains Function Updates Persistent State Information GetUserACL SetDataType SetQuota Consensus Invokes DataObjRepl Source: Reagan Moore Consistency Client Action Operation SysChksumDataObj

21 -based Data Management Implementation in irods Purpose (5 main types) Defines Collection DATA_ID DATA_REPL_NUM DATA_CHECKSUM SubType Archive Data grid Collection Digital Library Processing Pipeline Integrity Authenticity Access control Completeness Source: Reagan Moore Feature Correctness Defines Feature Consensus Property Defines Controls Procedure (11 default) Feature Consistency Replication Checksum Quota Data Type Feature Enforcement Points (70) Invokes Clients (50) SubType Periodic Assessment Criteria Digital Object Updates Workflow Chains Micro-service (317) Operation Updates Attribute Persistent State Information (338) msigetuseracl msisetdatatype msisetquota msidataobjrepl msisyschksumdataobj

22 Recap: -Based Data Management Purpose - reason a collection is assembled Properties - attributes needed to ensure the purpose Policies - enforce and maintain collection properties Procedures - functions that implement the policies Persistent state information - results of applying procedures Property assessment criteria validation that state information conforms to the desired purpose Federation - controlled sharing of logical name spaces These are the necessary elements for collection management 22

23

24 Default Policies in irods Data Grid 1. Setup a collection and trash directory for each account 2. Setup membership in public account 3. Manage deletion of account 4. Manage renaming of the data grid 5. Manage path permission checking 6. Manage resource quota 7. Manage use of parallel I/O streams for large files 8. Manage selection of default storage location 9. Manage selection of storage location for replication 10. Manage selection of number of processes to use when multitasking 11. Manage selection of physical path name

25 irods Rules: defining the policies Server-side workflows Action condition workflow chain recovery chain Condition - test on any attribute: Collection, file name, storage system, file type, user group, elapsed time, IRB approval flag, descriptive metadata Workflow chain: Micro-services / rules that are executed at the storage system Recovery chain: Micro-services / rules that are used to recover from errors 25

26 irods Micro-Services Function snippets that wrap a well-defined process Compute checksum Replicate file Integrity check Zoom image Get tiff image cutout Search PubMed Written in C or Python Recovery micro-services to handle failure Web services, external applications, can be wrapped as micro-services Can be chained to perform complex tasks Micro-services invoked by rule engine 26

27 irods Micro-Services Over 300 published microservices Pluggable: write, publish, re-use 27

28 Example: unified view of data idrop web client Spread across: 1) Disk-storage at UNC, 2) Disk-storage at RENCI, 3) Tape-storage at RENCI 28

29 Example: unified view of data 29

30 Example: data replication policy UNC Data Center RENCI Data Center Isilon E-iRODS icat Server E-iRODS Server DDN9900 StorNext Appliance Two working copies kept For data recovery and to allow analysis at both sites Tape Library Copy me and Data copied metadata control copy process Only on certain files (fastq, finished bam files) irods rule run nightly does the copy Performs copy, verifies copy successful, resets copy me attribute Versioning to allow for re-runs of patient samples 30

31 Example: data access policy Challenge Millions of files across different projects, growing daily Hundreds of users across different labs, changing frequently How to control access UNIX ACLs became too unwieldy Moving data means reproducing permission and group settings : access given if user and data belong to the same groups Tag data with group metadata (e.g., Lab X lung tumor study) Access rule: user s group must match data group E.g. (user y member of Lab X lung tumor study) Advantage: Data group tag generated as part of workflows, automatically Data can be moved without breaking permission model User-Data linkage not based on directory and file names Thanks to Sai Balu at LCCC 31

32 The Data Life Cycle - Collections Each data life cycle stage increases the value and usability of the original collection Project Collection Data Grid Data Processing Pipeline Digital Library Reference Collection Federation Private Shared Analyzed Published Preserved Sustained Local Distribution Service Description Representation Re-purposing Jeff gets data from a sensor Jeff shares data with colleagues Together w/ colleagues, analyzes data and produces results Results peerreviewed and published Jeff et. al. hit jackpot: collection now accepted as ref collection for decades Hydrology Datagrid grows in value to ecology and biology and federated

33 Lifecycles in an R&D data-driven ecosystem UNC STORAGE (Tape, Drives) UNC HPC RENCI STORAGE (Tape, Drives) RENCI HPC RENCI Hadoop Genomics Storage Genomics HPC Genomics Hadoop Lab Machines IT Machines Control over: Data movement and replication Metadata standards Archival, deletion, and retention Wild West As processes mature Policies as much control as needed irods Integration with workflows, hadoop, databases Hiding complexities Automation, all policy driven Analysts Data Providers Automated Processes External Partners Developers IT Staff, while transitioning adhoc practices to production processes

34 irods Clients APIs: Java, C, C++, Fortran, PHP, Python General Interfaces icommands UNIX and Windows command line interface idrop GUI interface idropweb web version of idrop interface Windows browser Web-DAV FUSE Parrot Domain specific clients: Grid tools (GridFTP, SAGA) Portals (EngineFrame) Web services (VOSpace, irods-rest) Workflows (Kepler, Taverna, NCSA Cyberintegrator) Digital libraries (Dspace, Fedora)

35 Storage Resources UNIX file system irods POSIX Driver Local Cache Universal Mass Storage System HPSS Tivoli Storage Manager Windows file system DBO SQL RDBMS HPSS Microservice Objects SRB DDN WOS Z39.50 HTTP FTP Amazon S3 Thredds HDFS/Hadoop

36 Pluggable Storage Resources irods Smart Pluggable Resource Resource 1 (e.g. high performance drive) Resource 2 (e.g. nfs drive) Resource 3 (e.g. archive) Resource 3a (cheap array of disks) Resource 3b (tape) Tree-based approach allows for extending horizontally and vertically Greater range of customized solutions: hierarchical storage management, load balancing, high availability, tailored interfaces with high performance storage environments,

37 -Managed Pluggable Resources E-iRODS Resource Wrapper Pluggable Resource Local drive Remote drive PEPs irods Rules Engine Resource-specific rules User develops pluggable resource Code inspection allows for autogenerated policy enforcement points (PEPs) Grid admin can then develop standard irods policy-enforcing rules specific to the resource Use Case Example: Pluggable resource by default replicates to ensure high availability irods rule informs resource to turn off high availability on ingested files tagged with Protected Health Information metadata

38 Lifecycles in an R&D ecosystem UNC STORAGE (Tape, Drives) RENCI STORAGE (Tape, Drives) Genomics Storage Lab Machines UNC HPC RENCI HPC RENCI Hadoop Genomics HPC Genomics Hadoop IT Machines Wild West As processes mature NFS Hadoop DDN WOS RDBMS Programmatic APIs irods policy control Data Services Data Workflows Web services irods Clients pluggable Analysts Data Providers Automated Processes External Partners IT Staff

39 irods in clinical and translational research Presentation title goes here 39

40 Secure Medical Workspace Combines Virtualization, Endpoint Data Leakage Protection (DLP), standard security such as use of VPNs, network sniffing, antivirus, group policies, 40

41 Secure Access to Data on the Clinical Side Research Systems Clinician Researcher irods-enabled samtools 1) 4) 5) E-iRODS Portal Sequence Data 3) 2) Data Sets Secure Medical Workspace NCGenes EMR 1) Clinician request for sequence reads on patient X 2) Patient id lookup to obtain subject id 3) Subject id lookup in E-iRODS 4) Data sets packaged in zip file and retrieved 5) Data unzipped and displayed within secure workspace Clinical Studies Clinical Systems

42 Loosely coupled distributed SMWs DW Deduce irods client Research Workspace irods Data Server Research Workspace irods Data Server irods client I2b2 irods Data Catalogue DW Research Workspace SAS irods client irods Data Server Researchers can access data via local clinical information system (CIS) or as shared files. Sharing between sites is managed by a combination of CIS federation and data grid middleware (more flexible, less CIS lockin)

43 Questions? Presentation title goes here 43