Next Generation Sequencing

Transcription

1 Next Generation Sequencing Cavan Reilly December 5, 2012

2 Table of contents Next generation sequencing NGS and microarrays Study design Quality assessment Burrows Wheeler transform BWT example Introduction to UNIX-like operating systems Installing programs Bowtie SAMtools RNA-Seq Aligning reads to the transcriptome TopHat Cufflinks RNA-Seq using TopHat and Cufflinks RNAseq and count models DNA-Seq BCFtools

3 Next generation sequencing Over the last 5 years or so there has been rapid development of methods for next generation sequencing. Here is the process for the Illumina technology (one of the 3 major producers of platforms for next generation sequencing). The biological sample (e.g. a sample of mrna molecules) is first randomly fragmented into short molecules. Then the ends of the fragments are adenylated and adaptor oligos are attached to the ends.

4 Next generation sequencing The fragments are then size selected, purified and put on a flow cell. An Illumina flow cell has 8 lanes and is covered with other oligonucleotides that bind to the adaptors that have been ligated to the fragmented nucleotide molecules from the sample. The bound fragments are then extended to make copies and these copies bind to the surface of the flow cell. This is continued until there are many copies of the original fragment resulting in a collection of hundreds of millions of clusters.

5

6

7 Next generation sequencing The reverse strands are then cleaved off and washed away and sequencing primer is hybridized to the bound DNA. The individual clusters are then sequenced in parallel, base by base, by hybridizing fluorescently labeled nucleotides. After each round of extension of the nucleotides a laser excites all of the clusters and a read is made of the base that was just added at each cluster. If a very short sequence is bound to the flow cell it is possible that the machine will sequence the adaptor sequence-this is referred to as adaptor contamination. There is also a measure of the quality of the read that is saved along with the read itself.

8 Next generation sequencing These quality measures are on the PHRED scale, so if there is an estimated probability of an error of p, the PHRED based score is 10 log 10 p. If we fragment someone s DNA we can then sequence the fragments, and if we can then put the fragments back together we can then get the sequence of that person s genome or transcriptome. This would allow us to determine what alleles this subject has at every locus that displays variation among humans (e.g. SNPs).

9 Next generation sequencing There are a number of popular algorithms for putting all of the fragments back together: BWA, Maq, SOAP, ELAND and Bowtie. We ll discuss Bowtie in some detail later (BWA uses the same ideas as Bowtie). ELAND is a proprietary algorithm from Illumina and Maq and SOAP use hash tables and are considerably slower than Bowtie.

10 Applications There are many applications of this basic idea: 1. resequencing (DNA-seq) 2. gene expression (RNA-seq) 3. mirna discovery and quantitation 4. DNA methylation studies 5. ChIP-seq studies 6. metagenomics We will focus on resequencing (and SNP calling) and gene expression in this course.

11 NGS and microarrays Some have talked about the death of microarrays due to the use of NGS in the context of gene expression studies, but this hasn t happened yet. Compared to microarrays, RNA-seq 1. higher sensitivity 2. higher dynamic range 3. lower technical variation 4. don t need a sequenced genome (but it helps, a lot) 5. more information about different exon uses (more than 90% of human genes with more than 1 exon have alternative isoforms)

12 NGS and microarrays In one head to head comparison, 30% more genes are found to be differentially expressed with the same FDR (Marioni, Mason, Mane, et al. (2008)). These authors also found that the correlation between normalized intensity values from a microarray had a correlation of with the logarithm of the counts (this is similar to what others have reported). The largest differences occurred when the read count was low and the microarray intensity was high which probably reflects cross-hybridization in the microarray.

13 NGS and microarrays Others have found slightly higher correlations, here is a figure from a study by Su, Li, Chen, et al. (2011).

14 Study design Before addressing the technical details, we will outline some considerations regarding study design specific to NGS technology. There are 3 major manufacturers of the technology necessary to generate short reads, and they all have different platforms with resulting differences in the data and processing methods. These vendors are 1. Illumina, formerly known as Solexa sequencing 2. Roche, which owns 454 Life Sciences 3. Applied Biosciences and their SOLiD platform

15 Study design We will focus on the Illumina technology as the University of Minnesota has invested in this platform (the HiSeq 2000 platform is available in the Biomedical Genomics Center). We also have some capabilities to do 454 sequencing here. This platform produces longer reads (up to several hundred and growing), and is very popular in the microbiomics literature.

16 Study design By sequencing larger numbers of fragments one can estimate the sequence more accurately. The goal of using high sequence coverage is that by using more fragments it is more likely that one fragment will cover any random location in the genome. Clearly one wants to cover the entire genome if the goal is to sequence it, hence the depth of coverage is how many overlapping fragments will cover a random location. By deep sequencing, we mean coverage of 30X to 40X, whereas coverage of 4X or so would be considered low coverage.

17 Study design The rule for determining coverage is: coverage = (length of reads number of reads)/ (length of genome) For a given sequencing budget, there is a tradeoff between the depth of coverage and the number of subjects one can sequence. The objectives of the study should be considered when determining the depth of coverage. If the goal is to accurately sequence a tumor cell then deep coverage is appropriate, however if the goal is to identify rare variants, then lower coverage is acceptable.

18 Study design For example, the 1000 Genomes project is using about 4X coverage. A reasonable rule of thumb is that one probably needs 20-30X coverage to get false negatives less than 1% in nonrepeat regions of a duploid genome (Li, Ruan and Durbin, 2008). If the organism one is sequencing does not have a sequenced genome, the calculations are considerably more complex and one would want greater coverage in this case. Throughout this treatment we will suppose that the organism under investigation has a sequenced genome.

19 Study design Another choice when designing your study regards the use of mate-pairs. With the Illumina system one can sequence both ends of a fragment simultaneously. This greatly improves the accuracy of the method and is highly recommended. If the goal is to characterize copy number variations then getting paired end sequences is crucial.

20 Quality assessment We can use the ShortRead package in R to conduct quality assessment and get rid of data that doesn t meet quality criteria. To explore this we will use some data from dairy cows. You can find a description of the data along with the actual data in sra format at this site. While the NCBI uses the sra format to store data, the standard format for storing data that arises from a sequencing experiment is the fastq format. We can convert these files to fastq files using a number of tools-here is how to use one supported at NCBI called fastq-dump.

21 Quality assessment To use this you need to download the compressed file, unpack it and then enter the directory that holds the executable and issue the command (more on these steps later). sratoolkit centos linux64/bin/fastq-dump SRR sra Then this will generate fastq files which hold the data. If we open the fastq files we see they hold identifiers and information about the reads followed by the actual reads:

22 Quality HWI-EAS283:8:1:9:19 length=36 ACGAGCTGANGCGCCGCGGGAAGAGGGCCTACGGGG +SRR HWI-EAS283:8:1:9:19 length=36 HWI-EAS283:8:1:9:1628 length=36 CCTAATAATNTTTTCCTTTACCCTCTCCACATTAAT +SRR HWI-EAS283:8:1:9:1628 HWI-EAS283:8:1:9:198 length=36 ATTGAATTTGAGTGTAAACATTCACATAAAGAGAGA +SRR HWI-EAS283:8:1:9:198 length=36 The string of letters and symbols encodes quality information for each base in the sequence.

23 Quality assessment Then we define the directory where the data is and read in all of the filenames that are there > library(shortread) > datadir <- "/home/cavan/documents/ngs/bovinerna-seq" > fastqdir <- file.path(datadir, "fastq") > fls <- list.files(fastqdir, "fastq$", full=true) > names(fls) <- sub(".fastq", "", basename(fls))

24 Quality assessment Then we apply the quality assessment function, qa, to each of the files after we read it into R using the readfastq function > qas <- lapply(seq along(fls), function(i, fls) + qa(readfastq(fls[i]), names(fls)[i]), fls) This can take a long time as it has to read all of these files into R, but note that we do not attempt to read the contents of the files into the R session simultaneously (a couple of objects generated by the call to readfastq stored inside R will really bog it down). Then we rbind the quality assessments together and save to an external location. > qa <- do.call(rbind, qas) > save(qa, file=file.path("/home/cavan/documents/ngs/bovinerna-seq","qa.rda")) > report(qa, "/home/cavan/documents/ngs/bovinerna-seq/reportdir") [1] "/home/cavan/documents/ngs/bovinerna-seq/reportdir/index.html"

25 Quality assessment This generates a directory called reportdir and this directory will hold images and an html file that you can open with a browser to inspect the quality report. So we now use the browser to do this-we see the data from run 12 is of considerably higher quality than the others but nothing is too bad. One can implement soft trimming of the sequences to get rid of basecalls that are of poor quality. A function to do this is as follows (thanks to Jeremy Leipzig who posted this on his blog). Note that this can require a lot of computational resources for each fastq file.

26 Quality assessment # softtrim # trim first position lower than minquality and all subsequent positions # omit sequences that after trimming are shorter than minlength # left trim to firstbase, (1 implies no left trim) # input: ShortReadQ reads # integer minquality # integer firstbase # integer minlength # output: ShortReadQ trimmed reads library("shortread") softtrim <- function(reads, minquality, firstbase=1, minlength=5){ qualmat <- as(fastqquality(quality(quality(reads))), matrix ) quallist <-split(qualmat, row(qualmat)) ends <- as.integer(lapply(quallist, function(x){which(x < minquality)[1]-1})) # length=end-start+1, so set start to no more than length+1 to avoid # negative-length starts <-as.integer(lapply(ends,function(x){min(x+1,firstbase)})) # use whatever QualityScore subclass is sent newq <-ShortReadQ(sread=subseq(sread(reads),start=starts,end=ends), quality=new(class=class(quality(reads)), quality=subseq(quality(quality(reads)), start=starts,end=ends)),id=id(reads))

27 Quality assessment } # apply minlength using srfilter lengthcutoff <- srfilter(function(x) { width(x)>=minlength}, name="length cutoff") newq[lengthcutoff(newq)] To use: > library("shortread") > source("softtrimfunction.r") > reads <- readfastq("myreads.fq") > trimmedreads <- softtrim(reads=reads,minquality=5,firstbase=4,minlength=3) > writefastq(trimmedreads,file="trimmed.fq")

28 Bowtie background The keys to understanding how the algorithm implemented by Bowtie are the Burrows Wheeler transform (BWT) and the FM index. 1. the FM index: see Ferragina, P. and Manzini, G. (2000), Opportunistic data structures with applications 2. the Burrows Wheeler transform (BWT): see Burrows, M. and Wheeler, D.J. (1994), A Block-sorting lossless data compression algorithm. The Burrows Wheeler transform is an invertable transformation of a string (i.e. a sequence of letters).

29 Computing the BWT To compute this transform, one 1. determines all rotations of the string 2. sorts the rotations lexicographically to generate an array M 3. saves the last letter of each sorted string in addition to the row of M that corresponds to the original string.

30 Computing the BWT: an example Some descriptions of the algorithm introduce a special character (e.g. $) that is lexicographically smaller than all other letters, then append this letter to the string before starting the rest of the transform. This just guarantees that the first row of M is the original string so that one needn t store the row of M that is the original string. To see how the BWT works, let s consider the consensus sequence just upstream of the transcription start site in humans (from Zhang 1998): GCCACC We will use the modification that introduces the special character.

31 BWT example First add the special character to the end of the sequence we want to transform, denoted S with length N. GCCACC$ then get all of the rotations: $GCCACC C$GCCAC CC$GCCA ACC$GCC CACC$GC CCACC$G GCCACC$

32 BWT example then sort them (i.e. alphabetize these sequences) $GCCACC ACC$GCC C$GCCAC CACC$GC CC$GCCA CCACC$G GCCACC$ so the BWT is just the last column, typically denoted L, and we will refer to element i of this array as L[i] for i = 1,..., N and ranges of elements via L[1,..., k]. L=CCCCAG$

33 BWT example Importantly one can invert this transformation of the string to obtain the original string. This was first developed in the context of data compression, so the idea was to transform the data, then compress it (which can be done very efficiently after applying this transformation), then one can invert the transformation after uncompressing.

34 BWT example Suppose we have the BWT of a genome and want to count the number of occurrences of some pattern, e.g. AC, our query sequence. We can do this as follows. First, create a table that for each letter has the count of the number of occurrences of letters that are lexically smaller than that letter. For us this is just x $ A C G C(x)

35 BWT example Next we need to be able to determine the number of times that a letter x occurs in L[1,..., k] for any k, which we will denote O(x, k). Note that all of these calculations can be done once and stored for future use. Now search for the last letter of our query sequence, which is C, over the range of rows given by [C(x) + 1,..., C(x + 1)] where x is C so the range is [3,6]. Note that these are the rows of M that start with a C although we don t actually need to store all of M to determine this (it just follows from the alphabetization).

36 BWT example Then process the second to last letter, which for our example is A. To determine the range of rows of M to search we look at [C(x) + O(x, s 1 1) + 1,..., C(x) + O(x, s 2 )] where x is A and s 1 and s 2 are the endpoints of the interval from the previous step. Now and O(A, 3 1) = O(A, 2) = 0 O(A, 6) = 1 so the interval is [ ,..., 1 + 1] = [2, 2] then since this is the end of the query sequence we determine the number of occurrences by looking at the size of the range of this interval, which is here just 1.

37 BWT example If the range becomes empty or if the lower end exceeds the upper end the query sequence is not present. For example if we query on CCC and proceed as previously, the first step is the same (since this also ends in C), so then searching for C again we find [C(C) + O(C, s 1 1) + 1, C(x) + O(C, s 2 )] but O(C, 2) = 2 and O(C, 6) = 4 so the interval is

38 BWT example [ , 2 + 4] = [5, 6] which has size 2 (as there are 2 instances of CC in the sequence), so then proceed as in step 2 again to get which is [C(C) + O(C, 5 1) + 1, C(C) + O(C, 6)] [ , 2 + 4] = [7, 6] indicating that this sequence is not present.

39 BWT example For aligning short sequences to a genome we need to know where the short segments align, hence we need to determine the location of the start of the sequence in the genome. If the locations of some of the letters are known we can just trace back to those locations as follows. First, note that we know the first column of the array M. We know this due to 2 facts: 1. the first column will be strictly alphabetized 2. every letter in the sequence is represented in the last column L.

40 BWT example For our problem we have L=CCCCAG$ and so the first column is just F =$ACCCCG. As our sequence is of length N, then if we knew the permutation of 1,..., N that mapped F to L we would know the entire sequence (as we will see shortly). That one can determine this mapping is the key ingredient in computing the inverse BWT. Call this map T where we have F ( T (i) ) = L(i).

41 BWT example To find T, first consider M and a new array, call it M, that is obtained from M by taking the last element from each row and making it the first element but otherwise changing nothing. Here is M : C$GCCAC CACC$GC CC$GCCA CCACC$G ACC$GCC GCCACC$ $GCCACC

42 Inverting the BWT Note that every row of M has a corresponding row in M and like M M has all cyclic rotations of the original sequence S. Also note that the relative order of the rows with the same first letter are the same in both arrays: rows 3, 4, 5 and 6 in M become rows 1, 2, 3 and 4 in M. We can use this insight to determine the mapping from F to L, i.e. T. To determine T, we proceed sequentially as follows. L(1) = C and this is the first C in L so T (1) = i where F (i) is the first C in F so looking at F we see that i = 3.

43 BWT example Then continue with this thinking: T (2) = i and L(2) is the second C in L, but then looking at F we see that the second C in F occurs at position 4, so that T (2) = 4. Continuing in this fashion we find that T = (3, 4, 5, 6, 2, 7, 1).

44 BWT example We can then reconstruct the sequence in reverse order. We know the last element of the sequence is the last element of the first row of M which is the first element of L, which is here C. Then we proceed sequentially using the mapping T as follows S(N i) = L ( T i (1) ) where T i (x) = T i 1( T (x) ), i.e. repeatly apply the mapping T to itself.

45 BWT example so S(N 1) = L ( T (1) ) = L(3) = C S(N 2) = L ( T (3) ) = L(5) = A S(N 3) = L ( T (5) ) = L(2) = C S(N 4) = L ( T (2) ) = L(4) = C S(N 5) = L ( T (4) ) = L(6) = G If we were just trying to find the location of a particular match, like AC we could just stop at that point rather than recover the entire sequence.

46 FM index The FM index is a compressed version of L, C and O along with a compressed version of a set of indices with known locations in S. This allows one to trace back to a known location. So one could align short reads to a genome by first computing the BWT and compressing the results. Unfortunately many of the reads would not align due to sequencing errors and polymorphisms. Hence one needs to allow some discrepancies between the genome and the short reads. While this is adjustable, the default specifications for the program Bowtie are that there is a maximum of 2 mismatches in the first 28 bases and a maximum total quality score of 70 on the PHRED scale.

47 Introduction to Linux In order to use the latest tools in bioinformatics you need access to a linux based operating system (or Mac OS X). So we will give some background on how to use a linux based system. The linux operating system is based on the UNIX operating system (by a guy named Linus Torvalds), but is freely available (UNIX was developed by a private company). There are a number of implementations of linux that go by different names: e.g. Ubuntu, Debian, Fedora, and OPENsuse.

48 Introduction to Linux Most beginners choose Ubuntu (or one of its offshoots, like Mint). I use Fedora on my computers and centos is used on the server we will use for this course. You can install it for free on any PC. Such systems are based on a hierarchical structure for organizing files into directories. Much of the time we will use plain text files: these are files that have no formatting, unlike the files you use in a typical word processing program. To edit such files you need to use a text editor. I like vi and vim (which has capabilities to interact with large files).

49 Introduction to Linux The editor pico is simple to use and installed on the server we will use for this class. To use this you type pico followed by the file name-if the file doesn t exist it will create the file otherwise it will open up the file so that you can edit it. For example [cavanr@boris ~]$ pico temp1 will open the file temp1 for editing. Commands at the bottom of the window tell you how to conduct various operations.

50 Introduction to Linux You can create subdirectories in your home directory by typing ~]$ mkdir NGS This would create a NGS subdirectory where you can put files related to your NGS project. Then to keep your microarray data separate you could do [cavanr@boris ~]$ mkdir microarray To move around hierarchical directory structure you use the command cd. We enter the NGS directory with [cavanr@boris ~]$ cd NGS

51 Introduction to Linux To go back to our home location (i.e. the directory you start from when you log on) you just type cd and if you want to go up one level in the hierarchy you type cd... To list all of the files and directories in your current location you type ls. To copy a file you use cp, to move a file from one location to another you use mv and to remove a file you use rm. One can modify the behavior of these commands by using options.

52 Introduction to Linux For example, if I want to see the size of the files (and some additional information) rather than just list them I could issue the command [cavanr@boris ~]$ ls -l Other useful commands include more and less for viewing the contents of files and man for information about a command (from the manual). For instance [cavanr@boris ~]$ man ls Will open a help file with information about the options that are available for the ls command.

53 Introduction to Linux Wild cards are characters that operating systems use to hold a variable value. These are extremely useful as it lets the user specify many tasks without having to do a bunch of repetitive typing. For example, suppose you had a bunch of files that ended with.txt and you wanted to delete them. You could issue commands like [cavanr@boris ~]$ rm file1.txt [cavanr@boris ~]$ rm file2.txt [cavanr@boris ~]$ rm file3.txt... Or you could just use [cavanr@boris ~]$ rm file*.txt or even [cavanr@boris ~]$ rm *.txt

54 Introduction to Linux Most linux configurations will check that you really want to remove the files, but if you are removing many files, the process of typing yes repeatly is unnecessary. Just type ~]$ rm -f *.txt When you obtain programs from the internet that run on linux systems these programs will typically install many subdirectories, so if you want to get rid of all the contents of a directory, first enter that directory, [cavanr@boris ~]$ cd olddir

55 Introduction to Linux then issue the command olddir]$ rm -rf * then go back up one level [cavanr@boris olddir]$ cd.. and get rid of the directory using the rmdir command [cavanr@boris ~]$ rmdir olddir If you issue a command to run a program but wish you hadn t and want it to stop you push the Ctrl and C keys simultaneously.

56 Introduction to Linux To connect to a linux server from a Windows machine you need a Windows program that allows you to log on to a terminal and the ability to transfer files to and from the remote server and your Windows machine. WinSCP is a free program that allows one to do this. You can download the data from the WinSCP website and it installs like a regular Windows program: just allow your computer to install it and go with the default configuration. To get a terminal window you can use a program called PuTTY. So download the program PuTTY (which is just the file called PuTTY.exe) and create a subdirectory in your Program Files directory called PuTTY.

57 Introduction to Linux Then save PuTTY.exe in the directory you just created. Once installed start WinSCP and use boris.ccbr.umn.edu as the hostname along with your user name and password. You can then get a terminal window using putty: just click on the box that shows 2 computers that are connected right below the session tab. Then disconnect when you are done (using the option in the session tab).

58 Installation of Bowtie Recall: Bowtie is a program that aligns short reads to an indexed genome. Note that there is a beta release of a program called Bowtie 2 that allows gapped alignments, Bowtie just allows a couple of nucleotides to disagree between the read and the genome. To use Bowtie we need a set of short reads (in fastq format) and the index of a genome and of course the program itself. We will download a compressed version of the program (an executable) and after we uncompress it, we can use the executable, i.e. a program that does something. Bowtie has the capability to build indexes if we have the genome sequence.

59 Installation of Bowtie First set up a directory to hold all of this work [cavanr@boris ~]$ mkdir RNAseq then create a directory to hold all of the programs we will download and set up, we will call it bin (for binaries). [cavanr@boris ~]$ mkdir bin Then enter this directory so we can install some programs. [cavanr@boris ~]$ cd bin

60 Installation of Bowtie In order to know what version of software you need to install you need to check the architecture of the computer. To do so we issue the following command bin]$ uname -m x86 64 So this is a 64 bit machine (32 bit machines will report something like i386). So get the appropriate version of Bowtie using the wget command [cavanr@boris bin]$ wget 64.zip

61 Installation of Bowtie then unpack it bin]$ unzip bowtie linux-x86 64.zip Archive: bowtie linux-x86 64.zip creating: bowtie / creating: bowtie /scripts/ inflating: bowtie /scripts/build test.sh inflating: bowtie /scripts/make a thaliana tair.sh... This will generate a subdirectory with the name bowtie , so enter that subdirectory so that you can have access to the executable called bowtie in this subdirectory.

62 Aligning reads to an indexed genome The program ships with some E. coli data, so we will check the installation with this. To use the program, you type the name of the executable, then give the first part of the name of the files that hold the indexes and finally give the location of the set of reads. With the standard Bowtie installation there is a subdirectory called reads that has 1000 short sequences, so we are telling the program to look there for the data. So to test the installation type the following (on some linux installations you don t need to specify the./ before bowtie) [cavanr@boris bowtie ]$./bowtie e coli reads/e coli 1000.fq

63 Aligning reads to an indexed genome Then you get a bunch of output written to the screen and at the end you should see # reads processed: 1000 # reads with at least one reported alignment: 699 (69.90%) # reads that failed to align: 301 (30.10%) Reported 699 alignments to 1 output stream(s) to examine the output you can have Bowtie write it to file by putting a filename after the command (the program will generate this file if it doesn t already exist) [cavanr@boris bowtie ]$./bowtie e coli reads/e coli 1000.fq e coli.map then the file e coli.map will contain all of the stuff that has been written to screen.

64 Aligning reads to an indexed genome If you open e coli.map up you should see r0 - gi ref NC ATGCTGGAATGGCGATAGTTGGGTGGGTATCGTTC :9;;<===>?@@@@AAA ABCCCDE 0 32:T>G,34:G>A r1 - gi ref NC CGGATGATTTTTATCCCATGAGACATCCAGTTCGG :9;;<===>?@@@@AAA ABCCCDE 0 r2 - gi ref NC CATAAAGCAACAGTGTTATACTATAACAATTTTGA :9;;<===>?@@@@AAA ABCCCDE 0 r5 + gi ref NC CAGCATAAGTGGATATTCAAAGTTTTGCTGTTTTA EDCCCBAAAA@@@@?>===<;;9: r7 + gi ref NC GCATATTGCCAATTTTCGCTTCGGGGATCAGGCTA EDCCCBAAAA@@@@?>===<;;9: r8 + gi ref NC GGTTCAGTTCAGTATACGCCTTATCCGGCCTACGG EDCCCBAAAA@@@@?>===<;;9: :A>T,33:C>G r9 - gi ref NC GCCTGTTCGGCGTTGAGGGTAATGAAATCATCGCC :9;;<===>?@@@@AAA ABCCCDE 0

65 Aligning reads to an indexed genome To get a prebuilt index for some other organism, say S. cerevisiae you can go to the site ftp://ftp.cbcb.umd.edu/pub/data/bowtie indexes First get the yeast (i.e. S. cerevisiae) genome build from the Bowtie website (the CYGD version). [cavanr@boris bowtie ]$ wget ftp://ftp.cbcb.umd.edu/pub/data/bowtie indexes/s cerevisiae.ebwt.zip Then move this into the indexes directory (note I mistakenly downloaded it from the main bowtie subdirectory) and unpack it

66 Aligning reads to an indexed genome indexes]$ unzip s cerevisiae.ebwt.zip Archive: s cerevisiae.ebwt.zip inflating: s cerevisiae.1.ebwt inflating: s cerevisiae.2.ebwt inflating: s cerevisiae.3.ebwt... then move back to the main directory and issue the following command [cavanr@boris bowtie ]$./bowtie -c s cerevisiae ATTGTAGTTCGAGTAAGTAATGTGGGTTTG

67 Aligning reads to an indexed genome then you should see output like 0 + Scchr ATTGTAGTTCGAGTAAGTAATGTGGGTTTG IIIIIIIIIIIIIIIIIIIIIIIIIIIIII 0 # reads processed: 1 # reads with at least one reported alignment: 1 (100.00%) # reads that failed to align: 0 (0.00%) Reported 1 alignments to 1 output stream(s) What the command did was align the single sequence ATTGTAGTTCGAGTAAGTAATGTGGGTTTG against the yeast indexed genome. By using the -c option we mapped a single sequence from the command line to the genome-you would never want to do this except for testing.

68 Building a new index We will build an index from a FASTA formatted file for a plasmid from an E. coli strain that was responsible for a food poisoning incident in Japan. First download the sequence file NC fna [cavanr@boris bowtie ]$ wget ftp://ftp.ncbi.nlm.nih.gov/genomes/bacteria/escherichia coli O157 H7 Sakai uid57781/nc fna Here is the top of that file: it is a FASTA file. >gi ref NC Escherichia coli O157:H7 str. plasmid posak1, complete sequence Sakai TTCTTCTGCGAGTTCGTGCAGCTTCTCACACATGGTGGCCTGCTCGTCAGCATCGAGTGCGTCCAGTTTT TCGAGCAGCGTCAGGCTCTGGCTTTTTATGAATCCCGCCATGTTGAGTGCAGTTTGCTGCTGCTTGTTCA TCTTTCTGTTTTCTCCGTTCTGTCTGTCATCTGCGTCGTGTGATTATATCGCGCACCACTTTTCGACCGT

69 Building a new index Then issue the command to build the index: [cavanr@boris bowtie ]$./bowtie-build NC fna e coli O157 H7 which generates some index files [cavanr@boris bowtie ]$ ls e coli O157 H7.* e coli O157 H7.1.ebwt e coli O157 H7.4.ebwt e coli O157 H7.2.ebwt e coli O157 H7.rev.1.ebwt e coli O157 H7.3.ebwt e coli O157 H7.rev.2.ebwt

70 Building a new index Then move these to the indexes directory [cavanr@boris bowtie ]$ mv e coli O157 H7.* indexes then as with the yeast example, let s align a single read from the command line: [cavanr@boris bowtie ]$./bowtie -c e coli O157 H7 GCGTGAGCTATGAGAAAGCGCCACGCTTCC and if everything went as planned you should get 0 + gi ref NC GCGTGAGCTATGAGAAAGCGCCACGCTTCC IIIIIIIIIIIIIIIIIIIIIIIIIIIIII 0 # reads processed: 1 # reads with at least one reported alignment: 1 (100.00%) # reads that failed to align: 0 (0.00%) Reported 1 alignments to 1 output stream(s)

71 Altering your $PATH Note that we need to be in the same directory level as our executable in order to use that executable thus far-this is not necessary. To have access to an executable anywhere in your system you should alter your PATH variable in the linux environment to include where you keep your executables then write the executables to your path. So, let s first make the bin directory we created part of our path: [cavanr@boris ~]$ export PATH=$HOME/bin:$PATH

72 Altering your $PATH and copy the executables: note that by copying the name of the executable without the./ at the start we can just use the name of the executable, i.e. bowtie ]$ cp bowtie $HOME/bin/./bowtie bowtie ]$ cp bowtie-build $HOME/bin/./bowtie-build bowtie ]$ cp bowtie-inspect $HOME/bin/./bowtie-inspect

73 Altering your $PATH To test that this works let s go back to our home directory and issue our bowtie command with the 1000 sample E. coli reads (note that we need to tell Bowtie where to find the fastq and index files) [cavanr@boris ~]$ bowtie $HOME/bin/bowtie /indexes/e coli $HOME/bin/bowtie /reads/e coli 1000.fq and it works fine.

74 Producing SAM files Bowtie can also produce a special file type that is widely used in the NGS world, a SAM file. To do so you use the -S option in the call to bowtie (and use the.sam extension on the output file) [cavanr@boris bowtie ]$ bowtie -S e coli reads/e coli 10000snp.fq ec snp.sam Here is the top of the file ec VN:1.0 SN:gi ref NC ID:Bowtie VN: CL:"bowtie -S e coli reads/e coli 10000snp.fq ec snp.sam" r0 16 gi ref NC M * 0 0 GCAATGATAAAAGGAGTAACCTGTGAAAAAGATGC XA:i:0 MD:Z:35 NM:i:0 r1 0 gi ref NC M * 0 0 TATGTTGGCAATATTGATGAAGATGGCGTCTGCCG EDCCCBAAAA@@@@?>===<;;9: XA:i:0 MD:Z:35 NM:i:0

75 SAMtools The SAM (for sequence alignment/map) file type has a very strict format. These files can be processed in a highly efficient manner using a suite of programs called SAMtools. Here is how to setup these tools Download the latest version in your bin directory samtools tar.bz2. [cavanr@boris bin]$ wget

76 SAMtools then unpack bin]$ tar -jxvf samtools tar.bz2 then there is a directory full of C programs called samtools , so go in there and type [cavanr@boris samtools ]$ make and write the executable to your path [cavanr@boris samtools ]$ cp samtools $HOME/bin/./samtools

77 SAMtools example To use this for variant calling, we need an example with at least 2 subjects, so let s use the example that ships with SAMtools. We will return to this example latter, but for now we will just take a look at the pileup which represents the extent of coverage. Here is how to use SAMtools to index a FASTA file: [cavanr@boris samtools ]$ samtools faidx examples/ex1.fa

78 BAM and SAM files then create a BAM file from the SAM file (BAM files are binary, so they are compressed but not readable by eye) [cavanr@boris samtools ]$ samtools import examples/ex1.fa.fai examples/ex1.sam.gz ex1.bam should get [sam header read2] 2 sequences loaded

79 BAM and SAM files then index the BAM file samtools ]$ samtools index ex1.bam then we can view the alignment to visually examine the extent of coverage and the depth of coverage samtools ]$ samtools tview ex1.bam examples/ex1.fa and hit q to leave that visualization.

80 Visualization of the pileup CACTAGTGGCTCATTGTAAATGTGTGGTTTAACTCGTCCATGGCCCAGCATTAGGGAGCTGTGGACCCTGCAGCCTGGC A...C C T.T.T..T...T..TC C T......,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,......T T

81 RNA-Seq As was mentioned previously, we can use next generation sequencing to generate sequences from samples of RNA. The transcriptome is the collection of all transcripts that can be generated from a genome. Current estimates indicate that 92-94% of human transcripts with more than one exon have alternatively spliced isoforms (Wang, Sandberg, Luo, et al. Nature, 2008). This means that the same gene can give rise to multiple mrna molecules. Some will differ because they have have different exons assembled into an mrna, but other possibilities include differences in use of the transcription start site (TSS), and differences in the 5 and 3 untranslated regions (UTRs).

82 Aligning reads to the transcriptome Differences in exon usage will result in mrnas from the same gene leading to different proteins, hence we speak of differential coding DNA sequences (CDS). One of the differences that arises due to differential 3 UTR usage is alternative polyadenelation patterns. These differences in 5 and 3 UTR usage lead to mrna molecules that have differential affinity for proteins that bind to the mature mrna molecule and we are just starting to understand how biological processes are regulated by these subtle mechanisms.

83 RNA-Seq Note that there is a difference in aligning a set of reads from an RNA sample and a DNA sample: the RNA molecules will have undergone removal of introns hence they won t align directly to the genome. If there is an annotated transcriptome for an organism you should start out trying to use this to analyze your data. However most investigations find evidence for transcripts that are not currently annotated, even for well studied organisms like mice and humans.

84 Aligning reads to the transcriptome We will investigate the role of using an annotated transcriptome for a class project: we will split into 4 groups and each group will either not use or use an annotated transcriptome during the mapping stage and during the transcript assembly stage. So we will do a 2 factor complete factorial design: you guys will run the analysis and I will summarize the findings the last day of class. More details on this once we have discussed the details of how to conduct the analysis.

85 Duplicates One of the differences between using next generation sequencing for resequencing and profiling RNA levels is that with libraries prepared from RNA, one often finds duplicate reads. In fact it is not unusual to find that 60-80% of the reads are duplicates.

86 Duplicates In a resequencing study the chances of finding duplicates is quite low: for example, the human haploid genome is about 3 billion bps so the probability of observing a duplicate is about 1 in 3 billion, so with million reads, duplicate reads seem unlikely. Duplicate reads can arise as a PCR artifact, hence in resequencing studies duplicate reads should probably be treated as artifacts and removed. There is a SAMtools function called rmdup that can be used to do this, but read the manual on its use: for example, it is designed for paired end reads by default.

87 Library complexity However, when we consider RNA-Seq libraries, duplicates might arise just due to low complexity of our sample. By low complexity we mean there are not many distinct RNA molecules. The median length of a human transcript is 2186 nucleotides (according to the UCSC hg19 annotation), so if one has a library with only 1 type of transcript and generates 20 million reads from it one is guaranteed to find duplicates. So for analysis of RNA-Seq experiments it is probably unwise to remove duplicates.

88 Alignment algorithms There are a number of algorithms that are designed to align a set of short reads to the transcriptome, here is a short non-exhaustive list. 1. TopHat 2. GSNAP 3. MapSplice 4. SpliceMap 5. HMMSplicer However it is not clear at this point if any of these really dominate the others in terms of performance. However, TopHat is perhaps the most popular currently, and as it builds off of Bowtie, we will use this program in this course.

89 TopHat The basic idea behind TopHat is that if 2 segments of a read map to different locations on the genome then they are likely mapping to 2 different exons. To use TopHat you need fastq files for all of our samples and the genome sequence for the organism. Note that sometimes the data for a single sample will be in more than one fastq file. While we can use Bowtie to align short reads to a genome, reads that span an intron, or genes that span a splice junction, will not align directly to the genome.

90 TopHat If a read contains an internal segment that should map to an exon, Bowtie will fail to map this read to its correct genomic location as it won t match the genome outside of this exon. For this reason, Tophat initially splits the reads up into 25 base units, does the read mapping, then reassembles reads. Note: if you are using a set of reads that are shorter than 50 bases you should instruct TopHat to break the reads into fragments that are about half the length of your reads. For example, if you have a set of Illumina reads that are 36 bases you should instruct TopHat to break them into fragments of length 18 (using the --segment-length options, like --segment-length 18), otherwise it may not detect any splice junctions.

91 TopHat TopHat starts out by mapping the reads to the genome using Bowtie and sets aside those reads that don t map. It then constructs a set of disjoint covered regions (it no longer uses the program Maq as it did at the time of the publication): these are taken to be the initial set of exons observed in that sample. The program determines a consensus sequence using the reads, however in low coverage regions it will use the reference genome to determine the sequence. Also, as the ends of exons are likely only mapped by reads that span the splice junction, the disjoint regions covered by reads are expanded (45 bases by default) using the reference sequence.

92 TopHat For genes that are transcribed at low levels there could be gaps in the exons after the assembly step, hence exons that are very close are merged into a single exon. Introns of length less than 70 bases are rare in mammalian genomes, hence by default we combine exons that are separated by less than this many bases (this can be adjusted). The resulting set of putative exons (with their extensions from the reference genome) we will call islands. The 5 end of an intron is called a donor site and the 3 end is called the acceptor site.

93 TopHat This donor site almost always has the 2 bases GU at the end while the acceptor site is almost always AG, but these are sometimes GC-AG and AU-AC so current versions of TopHat also look for these acceptor and donor sites. TopHat uses these sequence characteristics to find potential donor and acceptor site pairs in the island sequences (not just adjacent ones, but nearby ones to allow for alternative splicing). It then looks at the set of reads that were initially unmapped and examines if any of these map to the sequence that spans the splice junction. To conduct this mapping TopHat uses the fact that most introns are between 70 and 500,000 bases (the maximum differs from the publication): these intron sizes are based on mammals, so the user should specify values for these if using a different organism.

94 TopHat It also requires that at least one read span at least 8 bases on each side of the splice junction by default, but this value can be adjusted. It also only uses the first 28 bases from the 5 end of the read to ensure that the read is of high quality, but this too can be adjusted. Finally the program reports a splice junction only if it occurs in more than 15% of the depth of coverage of the flanking exons (the 15% value comes from the Wang et al. 2008, Nature paper), but this is also adjustable as this is based on human data. If one supplies a transcriptome to TopHat via a GTF file, the program will first map reads to the transcriptome then try to identify splice junctions for the unmapped reads.

95 TopHat One can disable the subsequent search for splice junctions by specifying the -T option if you have provided a transcriptome. If you are going to use an annotated transcriptome then TopHat will need to construct an index for this and if you are processing multiple samples TopHat will redo exactly the same calculation for each. To save time, you should have TopHat do these calculations for the first sample and save the results. You can do this using the --transcriptome-index option. This option expects a directory and a name to use in this directory.

96 TopHat The exact syntax is like this: note that the --transcriptome-index option must be used right after the -G option. tophat -o out sample1 -G known genes.gtf \ --transcriptome-index=transcriptome data/known \ hg19 sample1 1.fq.z Then for processing the other samples we just tell TopHat where these indices are and don t specify the -G option tophat -o out sample2 \ --transcriptome-index=transcriptome data/known \ hg19 sample2 1.fq.z

97 TopHat To use the program first get the latest version bin]$ wget x86 64.tar.gz Then uncompress the binaries, enter the directory that is generated and copy the executables to your path. bin]$ tar zxvf tophat linux x86 64.tar.gz tophat linux x86 64]$ cp tophat $HOME/bin/./tophat tophat linux x86 64]$ cp * $HOME/bin/./

98 TopHat Then you should test the installation, so get some test data. tophat linux x86 64]$ wget data.tar.gz Then uncompress, go into that directory and run a test [cavanr@boris tophat linux x86 64]$ tar zxvf test data.tar.gz [cavanr@boris tophat linux x86 64]$ cd test data Then test by issuing the command [cavanr@boris test data]$ tophat -r 20 test ref reads 1.fq reads 2.fq

99 TopHat and if it is working correctly this yields the following [ :14:56] Beginning TopHat run (v2.0.6) [ :14:56] Checking for Bowtie Bowtie 2 not found, checking for older version.. Bowtie version: [ :14:56] Checking for Samtools Samtools version: [ :14:56] Checking for Bowtie index files [ :14:56] Checking for reference FASTA file [ :14:58] Run complete: 00:00:01 elapsed

100 Cufflinks While TopHat is useful for aligning a set of reads to the transriptome, one must use some other program to generate transcript level summaries of gene expression and test for differences between groups. One such program that is supported by the same group of researchers that develop TopHat is Cufflinks. Cufflinks uses the output from TopHat and we will see that one can use functionality of the R package cummerbund to examine the output and generate reports and publication quality graphs. It is designed for paired end reads but can be used for unpaired reads.

101 Cufflinks Cufflinks uses an expression for the likelihood of a set of paired end reads that depends on the length of the fragments: with paired end reads one can estimate the distribution of the lengths. Note that there is a fundamental identifiability problem with transcript assembly that is only mitigated by the presence of reads that span splice junctions. Suppose there are 2 exons and some reads map to both exons but we are unable to identify any reads that span splice junctions. If there are 2 exons there are 3 possible transcripts (2 with one of each exon and a third that uses both) so if we allow for all 3 transcripts there are an infinite collection of combinations of the 3 transcript levels that are consistent with observed read counts that hit both exons.

102 Cufflinks For this reason one must enforce a constraint on the transcripts: for example, we attempt to find the minimal set of transcripts that explain the exon counts. Cufflinks quantifies the level of expression of a transcript via the number of fragments per kilobase of exon per million fragments mapped (FPKM). This attempts to account for the fact that longer transcripts will generate more reads than shorter transcripts due to their length. Cuffdiff is a program that is bundled with Cufflinks that allows one to test for differences between experimental conditions.

103 Cufflinks The log of the ratio of FPKM values are approximately normal random variables: this allows one to define a test statistic once one determines an approximation to the standard error of this log ratio. This standard error depends on the number of isoforms for the transcript: if there is a single isoform there is no uncertainty in the expression estimate produced by the algorithm and the only source of variability is from biological replicates. If there are no replicates within conditions the standard error is computed by examining the variance across the 2 conditions. If there are not many transcripts that differ in their level of expression this will provide a reasonable standard error, but if there are replicates one can obtain better estimates of the variance of the log ratio by using the negative binomial distribution.

104 Cufflinks If there is only a single isoform then an approach similar to that used in the R package DESeq is used, but if there are multiple isoforms then uncertainty in the isoform expression estimates must be taken into account: this is done using a generalization of the negative binomial distribution called the beta negative binomial distribution. Some useful details: The program will not conduct significance tests with fewer than 3 replicates for each condition unless you explicitly instruct it to do so by specifying a different minimum with the --min-reps-for-js-test option. While both BAM and SAM files can be used for input the program requires that if chromosome names are present in records in the header of the file then the reads must be sorted in the same order.

105 RNA-Seq using TopHat and Cufflinks We ve already installed and tested the latest version of TopHat, so now let s install Cufflinks using what is now the familiar strategy: download the tarball, unpack it and write the executables to our path by copying them to bin. [cavanr@boris bin]$ wget x86 64.tar.gz [cavanr@boris bin]$ tar zxvf cufflinks linux x86 64.tar.gz [cavanr@boris bin]$ cd cufflinks linux x86 64 [cavanr@boris cufflinks linux x86 64]$ cp * $HOME/bin/./