Accelerating Life Science Discovery using a High-Performance Analytics Platform in a Collaborative Environment Overview

Transcription

1 Accelerating Life Science Discovery using a High-Performance Analytics Platform in a Collaborative Environment Overview October 7, 2015 Kathy Tzeng, PhD Worldwide Technical Lead Healthcare & Life Sciences IBM Systems Group

2 Genomic Solution Enablement Team Mission: Porting and Optimization of Genomics/Translational applications on IBM solution Developing Solutions with Partners Making IBM SW/HW available to Software developers Members: Independent Software Vendor (ISV) team Toronto Compiler Lab Boeblingen Development Lab Tokyo Research Lab Austin Research Lab 2

3 GENOMIC MEDICINE from Sequencing to Personalized Healthcare NHGRI, a branch of NIH, has defined 5 steps for genomic medicine. (source: E. Green et al., Nature 470, ) 3 Next Generation Sequencing (or other ingestion) the focus is on very large data generation, mainly from $1000 whole genome sequencing, and the data processing and reduction includes human, plant, animal, and microbiome genomics Translational Research/Early Discovery the focus is on data integration including genomic data, and the analytics required to identify biomarkers, understand disease mechanisms, and to identify new medical treatments Personalized Healthcare/Clinical Genomics the focus is on delivering genomic medicine to patients to improve outcomes by associating patients with known genomic specific treatments

4 A Computationally Challenging Problem Breakthroughs in Genomic Medicine require quantifying associations between known population traits, environmental factors, and biological responses Known Traits or Environmental Features Predictive Response Function Measured Biological Response F(t) W(t) R(t) Quantities describing population traits or environmental factors at time t Model of associations between features and responses as a function of time t Computational Challenges Feature combinatorics Large file sizes Large population sizes Unstructured data types Quantities describing response events for an organism at time t 4

5 Workload Challenge #1: Big Data Analytics Variant information requires a computationally intensive analysis of raw sequence data across thousands of genomic samples Processing time per genome 1 to 100 hours * on 1 compute node * Duration depends on selection of analytical tools and hardware High-Throughput Sequencing Assembly & Alignment Variant Calling Variant Annotations Raw Reads De Novo Assembly SOAPdenovo Velvet Reference-Based Mapping BWA Bowtie SOAP Reference Genomes TGCA GEO dbsnp Variant Calling Picard GATK SAMtools SOAPsnp Annotation Tools ANNOVAR Gene Ontology File Format FastQ BAM VCF Sample: intergenic SNP in IL23R associated with Crohn's disease 3 billion DNA base pairs Whole Human 30 x coverage ~ 150 GB (compressed) ~ 150 GB 100 to 200 MB 500 MB Each human genome can have a few million variants 5

6 Workload Challenge #2: Unstructured Information Scientific data must be extracted from very large volumes of natural language content, biomedical images, and other unstructured data, and transformed into a structured format for analysis Omics Data Variant Databases exonic NOD2 16 a frameshift SNP exonic GJB2 13 associated with hearing loss exonic CRYL1,GJB6 13 a 342kb deletion Phenotypic Data Ex. Clinical Histories, Medical Images was in good health until 2-3 months ago when she gradually developed fatigue and intermittent epigastric pain, Scientific Literature Peer-Reviewed Articles, Clinical Guidelines, Textbooks, Patents Information must be transformed into normalized structured data for statistical analysis and relationship visualization 6

7 Workload Challenge #3: Big Data Integration Discovery of genotype-phenotype associations requires an analysis of complex data types that must be integrated within a common analytical environment 1 Omics Data ##FORMAT=<ID=DP, ##FORMAT=<ID=HQ, #CHROM POS ID REF ALT rs G A Variant Calls & Annotations 2 Phenotypic Data 3 Knowledge Base Clinical Features, Environmental Factors, Biological Responses Electronic Text & Web Sites + Big Data Warehouse Environment Patient-Centric Logical Data Model Genotypic Data 1 Variant List VCF Patient ID Patient ID Observed Traits & Responses 2 Phenotypic Data Variant ID Phenotype ID Detail on a Single Variant Patient Population Observation Detail 3 Knowledge Base RDBMS and/or NoSQL 7

8 Key Capabilities Leading biomedical research organizations are asking for technology capabilities that will give them a low-cost solution to accelerate scientific discovery in Genomic Medicine Flexible, scalable, and low-cost high-performance compute and storage solutions capable of efficiently processing rapidly growing quantities of genomic and other types of complex life science data Seamless integration of complex life science data types on a common analytical platform Rapid extraction and analysis of unstructured language content from very large volumes of clinical and scientific documents Metadata collection capabilities providing detailed audit trails as source data are transformed into analytical results Tools for scientific collaboration that enable data and workload sharing tocross organizations and geographic boundaries in a secure environment that ensures data privacy 8

9 A Foundation for Computational Science IBM s Reference Architecture for Genomic Medicine supports big data computational research on a foundation of HPC compute, storage, and workload management capabilities Performance optimization for open source and commercial analytics applications Research Applications Computational Modeling Genomic Analysis Pipelines Text Analytics /NLP -Apache UIMA -IBM System T LAN Image Analysis Text Analytics for the conversion of natural language concepts into structured data entities IBM Research, IBM Watson, IBM Business Partners RDBMS or NoSQL database environments enabling rapid processing of large volumes of complex highdimensional data structures in a data warehouse IBM BigInsights, IBM Business Partners Big Data Foundation Big Data Warehouse + Workload Orchestration with Metadata Capture Data Management: File System & Storage / ILM Intelligent resource allocation, sharing, and monitoring across parallel HPC workloads IBM Platform Computing, IBM Business Partners Low-cost, low-latency, easy-access storage & archiving of data and metadata across heterogeneous environments IBM Spectrum Scale / Elastic Storage Server 9

10 IBM Systems Facilitate Scientific Collaboration Data management and analytics tools can be accessed and shared across heterogeneous systems in on-premise and cloud environments Local Data Center On-Premise Users External Collaborators (Heterogeneous Environments) Private Cloud Users Public Cloud Users 1/10 GbE Applications Big Data Warehouse HPC Network Workload Orchestration with Metadata Capture Data Management: File System / Storage ILM Workload Burst WAN 10GbE or InfiniBand Big Data foundation enables data access, data management, and HPC workload orchestration across heterogeneous onpremise, private cloud, public cloud, and hybrid cloud environments On-Premise Cluster Virtual Private Clouds Encrypted VPN 10

11 Workload Orchestration Platforms Genomics Translational Personalized Healthcare Access AppCenter (PAC, Galaxy, DataBiology, Lab7) Application & Workflow File & Database Visualization System & Log Compute Orchestrator (ASC/EGO, LSF, Symphony, PPM) HPC Cluster Big Data Spark Cluster Openstack Docker Storage Datahub (Spectrum Scale, Zato, Nirvana) SSD/Flash FC/IB Attached Low-cost Storage HA/DR Storage Cloud Storage 11

12 A framework for NGS and HPC Systems Architecture Users HPC Platform Management Software Stack Suite Scale-out cluster Scale-up SMP Spectrum Scale ESS Active Archive TSM/LTFS/HPSS Devices 12

13 IBM Genomics Reference Architecture The IBM Reference Architecture is an ecosystem of data management and analytics tools developed by IBM and industry-leading commercial and open source software providers Edico Genome 13

14 BioBuilds Open Source Bioinformatics Open Source bioinformatics tools for research, commercial, and regulated environments. Turn-key: Pre-built binaries and complete build scripts enable easy deployment Optimized: POWER8 binaries provide the best performance for your hardware Ready for the Clinic: A single source for tools streamlining verification and audit Long Term Support: Community sponsorship and support contracts ensure ongoing support for tools 14

15 Open Source Application Portfolio in BioBuilds ALLPATHS-LG Bedtools Bfast BLAST (NCBI) Bowite Bowtie2 BWA Cufflinks FastQC Numpy PICARD PLINK Python SAMTools SOAP3-DP SOAPDenovo SQLite R Bioconductor FASTA Trinity SHRiMP Updated tools HMMER (LE) OpenSSL IGV irods RNAStar ISAAC TMAP SOAPaligner/soap2 Updated tools Bowtie2 HMMER Tabix BWA HTSeq Mothur TopHat Velvet/Oases OpenSSL 15 15

16 Optimization of GATK from Broad Institute IBM works with genomics leaders to improve performance of analytical workflows like GATK on IBM Power 8 Systems 16

17 Optimization of Broad s Best Practice Pipeline ~ 65X Whole Human Genome analysis done within a day ~ 150X Whole Exome analysis done in 3.45 hours Steps Intel Runtime* IBM Runtime BWA Samtools MarkDuplicates RealignTargets IndelRealigner BaseRecalibrator PrintReads+Index PreProcessiong Total HaplotypeCaller 2.03 Input Dataset: G15512.HCC1954.1, coverage: 65x Both IBM and Intel solution: # of Machines = 1 # of cores/machine = 24 IBM Solution: GHz Power8 with GPFS Total Note*: 17

18 Performance of L3 Bioinformatics BALSA on Power 8 with GPU Power GHz, 2x k40 GPU and GPFS 18

19 IO Cache Library to Optimize Performance of Genomics Application IBM uses a File Cache Library to improve I/O Performance and reduce workflow runtimes Application: Illumina s Casava V. 1.8 (BCL to FASTQ) Data Set: 8 lanes of HiSeq data Without cache library With cache library Elapsed Time = 1730 min Elapsed Time = 107 min 19

20 Accelerating Genomics Applications using GPFS IBM and BIOVIA s Pipeline Pilot scale genomic analysis from the desktop to the enterprise using IBM GPFS Speed of the file system matters Bowtie2: NGS Benchmarks on 2.6 GHz idataplex with GPFS and NFS Elapsed Time in Minutes, lower is better GPFS NFS

21 Genomic Workflow Optimization Typical Genomic Sequencing Workflow Command Line bwa aln -t 12 -l 40 -n 3 -k 2 bwa sampe -a 700 -P -o 1000 samtools view bt samtools sort Picard: java Xmx8g -Djava.io.tmpdir MarkDuplicates.jar METRICS_FILE=metrics CREATE_INDEX=true VALIDATION_STRINGENCY=LENIENT REMOVE_DUPLICATES=true ASSUME_SORTED=true TMP_DIR Picard: java -Xmx8g -Djava.io.tmpdir AddOrReplaceReadGroups.jar SORT_ORDER=coordinate RGID=sample_lane RGLB=sample RGPL=illumina RGPU=lane RGSM=sample RGCN=center_name CREATE_INDEX=True VALIDATION_STRINGENCY=LENIENT TMP_DIR Gatk lite: java -Xmx8g -Djava.io.tmpdir -T RealignerTargetCreator -nt 1 Gatk lite: java -Xmx8g -Djava.io.tmpdir -T IndelRealigner -targetintervals -known 1000G_biallelic.indels.hg19.vcf Picard: java -Xmx8g -Djava.io.tmpdir FixMateInformation.jar SO=coordinate VALIDATION_STRINGENCY=LENIENT CREATE_INDEX=true TMP_DIR Gatk lite: java -Xmx#{JAVA_REQMEM}g -Djava.io.tmpdir -T CountCovariates recalfile - knownsites:dbsnp,vcf /gpfs/gpfs1/genome/snp_indel_vcf/dbsnp_137.hg19.vcf -cov ReadGroupCovariate -cov QualityScoreCovariate -cov CycleCovariate -cov DinucCovariate Gatk lit: java -Xmx8g -Djava.io.tmpdir -T TableRecalibration -recalfile -smode SET_Q_ZERO - solid_nocall_strategy THROW_EXCEPTION -nback 7 --baq RECALCULATE Gatk lite:java -Xmx4g -jar $GATK_BIN/GenomeAnalysisTK.jar -glm BOTH -R $REFERENCE -T UnifiedGenotyper I recalibrated.bam 21

22 Genomic Workflow Optimization IBM Platform Process Manager facilitates genomic workflow execution 22

23 Genomic Workflow Optimization IBM Platform LSF workload scheduler is linked to the Process Manager and maximizes the utilization of HPC resources to improve workflow runtimes Data Set: 37x coverage of whole human genomes Workflow Input: 74 fastq.gz files, Workflow Output: Recalibrated Bam file Dependency steps = Using LSF bsub w option Runs 1 st Set 2 nd Set 3 rd Set 4 th Set Total Sets 1 set on 8 nodes hrs hrs 4 sets on 8 nodes hrs 20.9 hrs hrs hrs hrs 23

24 Data Compression Appliance Compression Algorithms gzip on Power 8 with FPGA board available now CRAM Compression ratio (lossless) On average 1:3 for fastq files 1:2 to 1:4 with respect to BAM files depending on the sequencing depth and other factors. (from FASTQ to compressed BAM ratio is 16X) Speed/throughput 2.5GB/s on average (200 GB fastq can be compressed in 80 second) Achieved beyond 10 times speed up using 12 cores (approximately 0.5GB/min) FPGA acceleration is ongoing. Pistoia compression contest was held in James Bonfieldof Sanger Institute won with 1:9 compression ratio and 0.1GB/min CRAM is released late 2012 to compress BAM file by EBI and accepted by Global Alliance of Genomics and Health. IBM is collaborating with Sanger Institute and EBI on improving compression for genomics data Samtools, Picard, CRAM Source:Baker M.,Nature Methods7, (2010) 24

25 .. >187_29_706_F3 T T >187_29_829_F3 T >187_29_858_F3 T > Enterprise Data Management IBM works with Lab7 to deliver data provenance with performance, reliability and security Experimental Design Sample Prep Sequencing Mapping Analysis Reporting Meta Analysis Sample LIMS User Experience Workflow Engine Federated Data Engine Pipeline Engine Sample Data Reference Attribute Sheet Pipeline Visualization/EDA Lab7 ESP Comprehensive software platform --- combines LIMS and informatics functionalities h Data provenance ---maintains continuous data provenance by: Tracking the history of samples, analyses, and results Providing detailed audit trails 9 Sequencing platform flexibility ---manages data generated from any sequencing platform IBM Power System Solution with GPFS and Platform LSF delivers: Superior compute infrastructure --- Superior performance, scalability & maximum throughput 8 Outstanding enterprise-grade reliability and security: Reliability, Availability and Serviceability (RAS) features help avoid unplanned downtime IBM Power Security and Compliance (PowerSC ) enables security compliance automation and includes reporting for compliance measurement and audit (HIPAA) 8 Total cost of ownership --- Very affordable compared to like-sized x86 systems 25

26 Data Provenance with Performance, Reliability and Security 26 Databiology for Enterprise Functional Architecture Databiology for Enterprise Scientific Samples Annotation Ontologies Shopping Basket Social Comments + Attachments + WF Integration 3 C s (Configure, Command, Collaborate) Portal API Custom Web Apps via API Compute and Storage Softlayer LSF GPFS Project Management Roles + Access Lifecycle Management Meta Information Financial + Resource Mgmt Task Management Transport DBE Download Manager DBE Multiprot S3, SCP, RSync, SFTP, FTP HTTP Applications Import Analysis Visualization Configuration Infrastructure Compute Storage Network Identity Management Instruments Logic Everything as an app: Scripts, Binaries, Pipelines, Workflow Management, Virtual Machines Version Control + Reproducible Data Provenance IBM Power System Solution with GPFS and Platform LSF delivers: Superior compute infrastructure --- Superior performance, scalability & maximum throughput 8 Outstanding enterprise-grade reliability and security: Reliability, Availability and Serviceability (RAS) features help avoid unplanned downtime IBM Power Security and Compliance (PowerSC ) enables security compliance automation and includes reporting for compliance measurement and audit (HIPAA) 8Total cost of ownership --- Very affordable compared to like-sized x86 systems Interface Information Management Orchestration SaaS + customer specific instances Central hub to manage all omics data and to orchestrate all activities Functionally rich and orientated on key steps in R&D life cycle Insight to Instrument with best in class applications Easy integration with existing environments Automatic data provenance and reporting Cost neutral deployment Gradual roll-out / Low risk

27 transmart - Optimized on Power8 and Spectrum Scale transmart associates genotypic & phenotypic data for complex analytics Watson Explorer extracts insight from scientific literature and data record and provides enrichment to transmart s analysis 27

28 transmart Power8 Deployment Architecture Users Application Browser HTTP Web Server (Apache2) Watson Analytics Server HTTP I2b2 Application Server transmart Solr Full Text index Watson Analytics JDBC Application Server PLINK GPFS PostgreSQL transmart DB JDBC (Tomcat 7) Quartz Job Call R Analytics Tools Gene Patterns Power8 28

29 Accelerate transmart ETL by Power8/Spectrum Scale Dataset TCGA_OV Simulation GSE32583 GSE13168 GSE1456 GSE15258 No. Records 5,789,632 40,774, ,724 1,203,282 3,600,555 4,702,050 29

30 Zato s Scalable Data Federation Solution for Healthcare and Genomics Data Spanning Data Centers in Parallel with a Single Pane of Glass for Clinical and Research Applications on Power 8 and GPFS Imaging Data Lab Results LAN LAN Electronic Health Record Data LAN Nursing Home Records VPN Microbiology Reports LAN Claims Data Radiology Reports LAN VPN VPN Internet Genomic Data Accepted Medical Knowledge NIH Data CDC Data NLM Data 30

31 Thank You 31 22

32 32 32

33 Noblis BioVelocity is Developed and Optimized on IBM s Power 8 33