Data Analysis & Management of High-throughput Sequencing Data. Quoclinh Nguyen Research Informatics Genomics Core / Medical Research Institute

Transcription

1 Data Analysis & Management of High-throughput Sequencing Data Quoclinh Nguyen Research Informatics Genomics Core / Medical Research Institute

2 Current Issues

3 Current Issues

4 The QSEQ file Number files per run: 120 x 8 x (1 2) = files Number of reads per run: ~ 40M x 8 x (1 2) = ~320M 640M Total nucleotides (BPs): 320M x 150 x (1 2) = ~48 96 billions

5 What am I going to do with my sequencing data?

6 Overview Infrastructure So<ware Novoalign Picard GATK SAMtools BowEe Cufflinks MACS RTA RTA Gigabit / drag- and- drop netowrk Prod: 400 nodes (64GB and 8GB) Isilon:100TB Very Fast NAS /rcgenomics (20 TBs) Dev: 4 nodes/64gb Pipelines Exome Seq RNA Seq ChiP Seq NAS/SAN 400TBs Archives SSH,FTP SMB,HTTP hrp://lims.csms.edu VM Server (Linux) LIMS(NGS/Microarray) Linux (VM & Physical) MySQL Oracle LAMP Analysis / Mining

7 Overview Infrastructure So<ware Novoalign Picard GATK SAMtools BowEe Cufflinks MACS Prod: 400 nodes (64GB and 8GB) Isilon:100TB Dev: 4 nodes/64gb (3 nodes x 75G)+(4 nodes x 65G) = 485Gs (3 nodes x 24CPU) + (4 nodes x 16) = 120CPUs Pipelines Exome Seq RNA Seq ChiP Seq NAS/SAN 400TBs Archives >300 nodes X 2 CPU = >600CPUs >300 nodes x 4G = > 1200Gs Fast NAS storage capacity ~ 100TBs Other storage devices ~ 400TBs

8 RNA-Seq Analysis Pipeline Phase 1: NGS data processing Phase 2: Basic Analysis Phase 3: Downstream Analysis input Binary Data BCL files Sample 1 BAM. Sample N BAM genes.expr hits.bam transcripts.expr transcripts.gif Raw Data QSEQ files BAM - > FASTQ Cuffcompare Mapped Reads SAM - >BAM TopHat Mapped reads BAM file Cuffdiff UCSC QC by samtools Cufflinks CLC BIO output clean BAM files genes/express files deliver results

9 Data Processing Phase 1: NGS data processing Binary Data BCL files Raw Data QSEQ files Mapped Reads SAM - >BAM QC by samtools QC / Data filtering Bin and remove indexes Remove adapters (if any) Remove duplicate Removal of non-mapped sequences Filter out read mapping to ribosomal RNA Percentage of ribosomal? QSEQ files -> Aligned files (BAM) Parallel processing (MPI) What is in the alignment files? clean BAM files

10 The BAM/SAM file BAM (Binary Alignment Mapping) SAM (Sequence Alignment Mapping) Standardized output for alignment Contains all required information for downstream analysis Query name Segment sequence Quality score Phred- base

11 RNA-Seq Analysis Pipeline Phase 1: NGS data processing Phase 2: Basic Analysis Phase 3: Downstream Analysis input Binary Data BCL files Sample 1 BAM. Sample N BAM genes.expr hits.bam transcripts.expr transcripts.gif Raw Data QSEQ files BAM - > FASTQ Cuffcompare Mapped Reads SAM - >BAM TopHat Mapped reads BAM file Cuffdiff UCSC QC by samtools Cufflinks CLC BIO output clean BAM files genes/express files deliver results

12 RNA-Seq Data Analysis Phase 2: Basic Analysis Sample 1 BAM. BAM - > FASTQ TopHat Mapped reads BAM file Cufflinks genes/express files Sample N BAM One BAM file per sample Tophat bowtie : align short reads splice junction identifier Cufflinks Uses annotation file to count transcripts and isoforms Output files Gene expression : gene.expr Transcript expression : transcripts.expr Transcript in GTF format : transcripts.gft

13 RNA-Seq Analysis Phase 2: Basic Analysis Sample 1 BAM. Sample N BAM Genes expression genes.expr BAM - > FASTQ TopHat Mapped reads BAM file transcriptions expression transcripts.expr Cufflinks genes/express files

14 RNA-Seq Analysis Phase 2: Basic Analysis Sample 1 BAM. Sample N BAM Genes expression genes.expr BAM - > FASTQ TopHat Mapped reads BAM file Cufflinks genes/express files Unit of measurement (FPKM/RPKM) FPKM: Fragments per kilobase per million mapped reads 1kb transcript with 1000 alignments in a sample of 10M reads (out of which 8 million reads can be mapped) will have FPKM = 1000/(1*8) = 125

15 RNA-Seq Analysis Pipeline Phase 1: NGS data processing Phase 2: Basic Analysis Phase 3: Downstream Analysis input Binary Data BCL files Sample 1 BAM. Sample N BAM genes.expr hits.bam transcripts.expr transcripts.gif Raw Data QSEQ files BAM - > FASTQ Cuffcompare Mapped Reads SAM - >BAM TopHat Mapped reads BAM file Cuffdiff UCSC QC by samtools Cufflinks CLC BIO output clean BAM files genes/express files deliver results

16 RNA-Seq Analysis Phase 3: Downstream Analysis genes.expr hits.bam Cuffcompare Cuffdiff UCSC CLC BIO transcripts.expr transcripts.gif Cuffcompare Compare your assembled transcript to a reference annotation Track cufflinks transcripts across multiple experiments (e.g across time course) cuffcompare -<options> Cuffdiff Part of cufflinks package Find significant changes in transcriptions splicing, and promoter use. Viewer & Annotation deliver results

17 Cuffdiff Sample Differential expression at the transcript isoform level and at the gene level

18 RNA-Seq Analysis Pipeline Phase 1: NGS data processing Phase 2: Basic Analysis Phase 3: Downstream Analysis input Binary Data BCL files Sample 1 BAM. Sample N BAM genes.expr hits.bam transcripts.expr transcripts.gif Raw Data QSEQ files BAM - > FASTQ Cuffcompare Mapped Reads SAM - >BAM TopHat Mapped reads BAM file Cuffdiff UCSC QC by samtools Cufflinks CLC BIO output clean BAM files genes/express files deliver results

19 ChiP-Seq Analysis Pipeline Phase 1: NGS data processing Phase 2: Basic Analysis/Peak calling Phase 3: Downstream Analysis input Binary Data BCL files Sample 1 BAM. Sample N BAM peaks.bed peaks.xls model.r summits.bed Raw Data QSEQ files novoalign CEAS Mapped Reads SAM - >BAM Mapped reads BAM file UCSC CLC BIO output QC by samtools clean BAM files MACS peaks raw results - MoEf discovery - RelaEonship to gene structure - Gene set analysis - DifferenEal profile analysis - Other advanced analysis MACS: Model-base Analysis ChiP-seq / CEAS: Cis-regulatory Element Annotation System

20 I can start my research now.

21 Overview Infrastructure So<ware Novoalign Picard GATK SAMtools BowEe Cufflinks MACS RTA / drag- and- drop Prod: 400 nodes (64GB and 8GB) Isilon:100TB Very Fast NAS /rcgenomics (20 TBs) Dev: 4 nodes/64gb Pipelines Exome Seq RNA Seq ChiP Seq NAS/SAN 900TBs Archives SSH,FTP SMB,HTTP hrp://lims.csms.edu VM Server (Linux) LIMS(NGS/Microarray) Linux (VM & Physical) MySQL Oracle LAMP Analysis / Mining

22 Data Management Data storage Understand NGS data Not all data equally important How data gets storage? User access LIMS (Laboratory InformaEcs Management System) Service Request Access data

23 Data Storage Raw Data ~2 5TB per run NAS QSEQ/FASTQ Intensities BaseCalls Results SNV, Indel, expr LIMS Fast NAS

24 We are

25 Access the NGS LIMS

26 NGS LIMS

27 NGS LIMS Add Sample

28 NGS LIMS Add Sample

29 NGS LIMS Add Library

30 NGS LIMS Create a FlowCell

31 NGS LIMS Get Results

32 NGS LIMS Get Results

33 Questions?