Sequencing Analysis Software User Guide

Transcription

1 Sequencing Analysis Software User Guide For Pipeline Version 1.5 and CASAVA Version 1.0 FOR RESEARCH USE ONLY ACGTACGTACGTACG TACGTACGTACGTACGTA A G T G AC CG C T AC CGT ILLUMINA PROPRIETARY Catalog # SY Part # Rev. A

2

3 Notice This document and its contents are proprietary to Illumina, Inc. and its affiliates ( Illumina ), and are intended solely for the contractual use of its customers and for no other purpose than to use the product described herein. This document and its contents shall not be used or distributed for any other purpose and/or otherwise communicated, disclosed, or reproduced in any way whatsoever without the prior written consent of Illumina, Inc. For the proper use of this product and/or all parts thereof, the instructions in this document must be strictly and explicitly followed by experienced personnel. All of the contents of this document must be fully read and understood prior to using the product or any of the parts thereof. FAILURE TO COMPLETELY READ AND FULLY UNDERSTAND AND FOLLOW ALL OF THE CONTENTS OF THIS DOCUMENT PRIOR TO USING THIS PRODUCT, OR PARTS THEREOF, MAY RESULT IN DAMAGE TO THE PRODUCT, OR PARTS THEREOF, AND INJURY TO ANY PERSONS USING THE SAME. RESTRICTIONS AND LIMITATION OF LIABILITY This document is provided as is, and Illumina assumes no responsibility for any typographical, technical or other inaccuracies in this document. Illumina reserves the right to periodically change information that is contained in this document and to make changes to the products, processes, or parts thereof described herein without notice. Illumina does not assume any liability arising out of the application or the use of any products, component parts, or software described herein. Illumina does not convey any license under its patent, trademark, copyright, or common-law rights nor the similar rights of others. Illumina further reserves the right to make any changes in any processes, products, or parts thereof, described herein without notice. While every effort has been made to make this document as complete and accurate as possible as of the publication date, no warranty of fitness is implied, nor does Illumina accept any liability for damages resulting from the information contained in this document. ILLUMINA MAKES NO REPRESENTATIONS, WARRANTIES, CONDITIONS, OR COVENANTS, EITHER EXPRESS OR IMPLIED (INCLUDING WITHOUT LIMITATION ANY EXPRESS OR IMPLIED WARRANTIES OR CONDITIONS OF FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, MERCHANTABILITY, DURABILITY, TITLE, OR RELATED TO THE PERFORMANCE OR NONPERFORMANCE OF ANY PRODUCT REFERENCED HEREIN OR PERFORMANCE OF ANY SERVICES REFERENCED HEREIN). This document may contain references to third-party sources of information, hardware or software, products or services, and/or third-party web sites (collectively the Third-Party Information ). Illumina does not control and is not responsible for any Third-Party Information, including, without limitation, the content, accuracy, copyright compliance, compatibility, performance, trustworthiness, legality, decency, links, or any other aspect of Third-Party Information. Reference to or inclusion of Third-Party Information in this document does not imply endorsement by Illumina of the Third-Party Information or of the third party in any way Illumina, Inc. All rights reserved. Illumina, illuminadx, Solexa, Making Sense Out of Life, Oligator, Sentrix, GoldenGate, GoldenGate Indexing, DASL, BeadArray, Array of Arrays, Infinium, BeadXpress, VeraCode, IntelliHyb, iselect, CSPro, and GenomeStudio are registered trademarks or trademarks of Illumina, Inc. All other brands and names contained herein are the property of their respective owners. Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide iii

4

5 Revision History Part Number Revision Letter Date A August A April A December A June A January 2008 Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide v

6

7 Table of Contents Notice iii Revision History v Table of Contents vii List of Figures xi List of Tables xiii Chapter 1 Overview Analysis Computing Systems What s New Reporting Problems Technical Assistance Chapter 2 Core Pipeline Concepts Analysis Modules Use of Modules Running the Pipeline Modules Understanding the Run Folder Run Folder Structure Images Folder Data Folder Run Folder Naming File Naming Configuration/Parameters Calibration and Input Parameters Quality Scoring Image Offsets Frequency Cross-Talk Matrix Phasing/Prephasing Estimates Sample Information Alignment Algorithms ELAND Algorithm Description Chapter 3 Using GOAT Starting with Image Analysis Invoking GOAT for Image Analysis Running a GOAT Image Analysis Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide vii

8 viii CHAPTER 2 Paired Reads Parallelization Switch Nohup Command Command Line Options for GOAT General Options GOAT Options Paired Reads Makefile Targets Chapter 4 Using Bustard Starting with Base Calling Invoking Bustard for Base Calling Running Pipeline Base Calling Starting with SCS Image Analysis Data Starting with IPAR Image Analysis Data Paired Reads Parallelization Switch Nohup Command Command Line Options for Bustard General Options Bustard Options Paired Reads Makefile Targets Chapter 5 Using GERALD for Sequence Alignment Running GERALD as a Standalone Program Standard GERALD Analysis GERALD Configuration File Lane-Specific Options Optional Parameters Paired-End Analysis Options GERALD Parameters ANALYSIS Variables Analysis Parameters Lane-by-Lane Parameters USE_BASES Option QCAL_SOURCE Option Make Option Rerunning the Analysis Building an SRF Archive ELAND Alignments Missing Bases in ELAND Using ANALYSIS eland_extended Using ANALYSIS eland_pair Using ANALYSIS eland_tag Using ANALYSIS eland_rna Preparing the Reference Genome Chapter 6 Pipeline Analysis Output Visual Analysis Summary Catalog # SY Part # Rev. A

9 ix Results Summary Cluster Intensity Error Rates Text-Based Analysis Results GERALD Output Files in Temp Folder Interpretation of Run Quality Summary.htm IVC.htm All.htm and Error.htm Chapter 7 Advanced Pipeline Usage Running Bustard as a Standalone Program Filtering Parameters Analysis of Multiplexed Sequencing Runs GERALD Analysis Split_on_index.py Script Chapter 8 Using CASAVA Use Cases CASAVA Workflow Hardware and Software Requirements Expected Performance Estimating Build Depth CASAVA Input Files Export.txt Files Run.conf.xml Pair.xml Config.xml Summary.htm Methods Duplicate Removal Final Set of Reads Allele Calling SNP Calling CASAVA Read Start Counting Method Running CASAVA Examples Results Directory Running Specific Use Cases DNA Sequencing Analysis for Large Genomes DNA Sequencing Analysis for Small Genomes RNA Sequencing CASAVA Output Files Build Directory Build Web Page CASAVA Build Appendix A Requirements and Software Installation for Pipeline System Requirements Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide

10 x CHAPTER 2 Network Infrastructure Analysis Computer Installation Prerequisites Setting Up Reporting Installing the Pipeline Software Compiling on Other Platforms Directory Setup Appendix B Analysis Output File Descriptions Output File Types Intensity Files Main Sequence Files from Bustard Optional Files from Bustard Efficiency Intermediate Output Data Files Output File Formats Configuration/Parameters File Format Params File Config.xml Files RunInfo.xml File Appendix C Using Parallelization in Pipeline Make Utilities Customizing Parallelization Parallelization Limitations Memory Limitations Appendix D Reference Files for Eland_rna and CASAVA Eland_rna Reference Files Eland_rna Genome Files Abundant Sequences Files Splice Junction Set Human Eland_rna Reference Files Mouse Eland_rna Reference Files Rat Eland_rna Reference Files CASAVA Reference Files CASAVA Genome Files Genome Size File Exon Coordinates Set Human CASAVA Reference Files Mouse CASAVA Reference Files Rat CASAVA Reference Files Generating Reference Files Getting Data Files Genome Sequence Files Abundant Sequence Files Splice Junction Set Genome Size Files Non-Redundant Exon Set Catalog # SY Part # Rev. A

11 List of Figures Figure 1 Three Steps of Data Analysis Figure 2 Data Analysis Workflow Figure 3 Phasing and Prephasing Figure 4 Pipeline Modules Figure 5 SCS Real Time Analysis Run Folder Directory Structure Figure 6 IPAR/Pipeline Run Folder Directory Structure Figure 7 Frequency Cross-Talk Matrix and Phasing File Locations Figure 8 CASAVA Workflow Figure 9 CASAVA Build Directory Figure 10 Build Web Page Figure 11 Summary.htm File Figure 12 SNPs Graphs in Home.html Figure 13 Statistics Graphs in Home.html Figure 14 Chromosome.snp.txt File Opened in Excel Figure 15 Chromosome_genes_count.txt File Opened in Excel Figure 16 Run Folder Structure and Output File Types Figure 17 UCSC Genome Bioinformatics Web Page Figure 18 Selected Genome Web Page (Human) Figure 19 Annotation Database Files Web Page, Files Figure 20 Index (Data set by chromosome) Web Page Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide xi

12 xii List of Figures Catalog # SY Part # Rev. A

13 List of Tables Table 1 Illumina Customer Support Contacts Table 2 File Naming Components Table 3 GERALD Configuration File Parameters Table 4 GERALD Configuration File Lane-Specific Options Table 5 GERALD Configuration File Optional Parameters Table 6 GERALD Configuration File Paired-End Analysis Options Table 7 ANALYSIS Variables Table 8 Analysis Parameters Table 9 Lane-by-Lane Parameters Table 10 USE_BASES Options Table 11 QCAL_SOURCE Variable Values Table 12 Parameters for ANALYSIS eland_extended Table 13 Parameters for ANALYSIS eland_pair Table 14 Parameters for ANALYSIS eland_rna Table 15 Example of Relative Orientation Statistics Table Table 16 Example of Insert Size Statistics Table Table 17 Example of Insert Statistics Table Table 18 Text-Based Analysis Results Table 19 Example of Lane Results Summary Table 20 Example of Expanded Lane Summary Table 21 Expected Performance for Typical CASAVA Projects Table 22 Data Volumes Per Experiment Table 23 Intermediate Output File Descriptions Table 24 Final Output File Formats Table 25 Intermediate Output File Formats Table 26 Human Genome Reference Files for Eland_rna Table 27 Mouse Genome Reference Files for Eland_rna Table 28 Rat Genome Reference Files for Eland_rna Table 29 Human Genome Reference Files for CASAVA for RNA Sequencing Table 30 Mouse Genome Reference Files for CASAVA for RNA Sequencing Table 31 Human Genome Reference Files for CASAVA for RNA Sequencing Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide xiii

14 xiv List of Tables Catalog # SY Part # Rev. A

15 Chapter 1 Overview Topics 2 Introduction 3 Genome Analyzer Pipeline Software 4 Pipeline Workflow 4 CASAVA Software 5 What s New 6 Reporting Problems 7 Technical Assistance Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide 1

16 2 CHAPTER 1 Overview Introduction This user guide documents the Genome Analyzer Pipeline Software and the CASAVA Software. The Genome Analyzer Pipeline Software performs offline data analysis of a sequencing run. The CASAVA Software package performs post-sequencing analysis of data from reads aligned to the reference genome by Pipeline. The basic functionalities of these modules are described below. Analysis of Sequencing Data After the Genome Analyzer generates the sequencing images, the data is analyzed in three steps: image analysis, base calling, and sequence analysis (Figure 1). 1. Image analysis Uses the raw TIF files to locate clusters on the image, and outputs the cluster intensity, X,Y positions, and an estimate of the noise for each cluster. The output from image analysis provides the input for base calling. 2. Base calling Uses cluster intensities and noise estimates to output the sequence of bases read from each cluster, a confidence level for each base, and whether the read passes filtering. 3. Sequence analysis Allows for alignment to a reference sequence and visualization of the result. Figure 1 Three Steps of Data Analysis Analysis Computing Systems The different analysis steps can be performed by different analysis computing systems: Sequencing Control Software (SCS) real time analysis, which runs on the Genome Analyzer instrument computer. SCS real time analysis performs real-time image analysis and base calling. The Genome Analyzer Pipeline Software (Pipeline), which runs on a Linux analysis server. Pipeline can perform off-line image analysis, base calling, and sequence analysis. NOTE With the launch of GA IIx, IPAR has become functionally obsolete. However, if you have not upgraded to SCS 2.4 or later you can still use IPAR 1.3 and older versions of SCS to produce intensity data for analysis in Pipeline 1.3 and later versions. Catalog # SY Part # Rev. A

17 3 NOTE IPAR, SCS, and Pipeline image analysis may yield slightly different results, due to minor variations in libraries and algorithms used. The differences are negligible compared to experimental variation. Figure 2 Data Analysis Workflow The standard workflow is to perform image analysis and base calling using SCS real time analysis, after which Pipeline performs alignment using the base calling results (Figure 2). Genome Analyzer Pipeline Software The Genome Analyzer Pipeline Software is a set of utilities designed to perform a complete offline data analysis of a sequencing run. It is supplied as source code and scripts. The output data produced by the Genome Analyzer Pipeline Software are stored in a hierarchical folder structure called the Run Folder. The Run Folder includes all data folders generated from the Genome Analyzer and the data analysis software. For a detailed description of the Run Folder structure, see Understanding the Run Folder on page 13. The Pipeline requires a Linux system with specific processing and data storage capacity. For specific requirements, see System Requirements on page 116. Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide

18 4 CHAPTER 1 Overview Pipeline Workflow The image data from a sequencing run are saved on the Genome Analyzer computer in a folder structure organized by lane and tile number. The data are transferred to a network location for analysis after the sequencing run is complete or by mirroring the data to the storage location while the run progresses. SCS real time analysis and IPAR also transfer their output data to a network location for analysis by Pipeline after the run is complete. The following is an overview of the Pipeline workflow. Installation 1. Install the Pipeline prerequisites on a suitable Linux system. See Installation Prerequisites on page Install the Pipeline software and compile the Pipeline using the make command. See Installing the Pipeline Software on page Set up the Instruments directory for parameters files. See Directory Setup on page 121. Running the Analysis 1. Navigate (via the command line) to the Run Folder location. 2. Create a configuration file that specifies what analysis should be done for each lane. See GERALD Parameters on page 51 and GERALD Configuration File on page Run a check on the Run Folder. See Running a GOAT Image Analysis on page Add command line options, generate the analysis folder, and corresponding makefiles. See Command Line Options for GOAT on page Change to the analysis directory and start your analysis by executing makefiles. Analysis Output 1. View the analysis results of your run. See Visual Analysis Summary on page 68 and Text-Based Analysis Results on page Interpret the run quality. See Interpretation of Run Quality on page 77. CASAVA Software The CASAVA Software v1.0 (CASAVA) provides analysis for three basic use cases: DNA Sequencing for large genomes. DNA Sequencing for small genomes (data sets). RNA Sequencing. All types of analysis take export.txt files from Pipeline as input and produce a set of allele calls for Single Nucleotide Polymorphisms (SNPs). In addition, RNA Sequencing analysis provides counts for exons, genes and splice junctions. Use and properties of CASAVA are explained in Chapter 8, Using CASAVA. Catalog # SY Part # Rev. A

19 5 What s New Important Changes in Pipeline v1.5 New quality tables supporting flow cell v4 and cluster generation chemistry v4. Support for quality tables for 1.4mm flow cell and cluster generation chemistry v2. Important Changes in Pipeline v1.4 Improved the estimation of the alignment scores of longer reads. Can start from SCS real time analysis base calling data. Bustard supports the binary intensity data format generated by SCS real time analysis (with the --CIF option). The format of the Firecrest output has changed. The intensity and noise files are now generated cycle by cycle. The pipeline uses the data from the file RunInfo.xml (normally generated by SCS) to identify the boundaries of the reads (including index reads). PhageAlign produces export files. Important Changes in Pipeline v1.3 The quality scoring scheme has changed to the Phred scoring scheme, encoded as an ASCII character by adding 64 to the Phred value. A Phred score of a base is: Q phred =-10 log 10 (e) where e is the estimated probability of a base being wrong. The Bustard output formats have changed; a new file format called "qseq.txt" is used to store read IDs, sequence and quality information as well as filter information. The old Bustard output formats can be produced optionally with the "-- with-seq", "--with-prb", "--with-siq2", "--with-qval" options. A new build system is used. The installation still involves installing the prerequisites and then typing "make" and "make install" in the top-level pipeline folder. The executables can now be found in the bin/ directory (e.g. bin/goat_pipeline.py). For base-call auto-calibration, the option "--with-qval" needs to be specified at the goat_pipeline.py or bustard.py command line. The Gerald analysis modes "expression" and "eland" are deprecated. They are replaced by "eland_tag" and "eland_extended" respectively. The "CONTAM_FILE" feature in PhageAlign mode is deprecated. Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide

20 6 CHAPTER 1 Overview Reporting Problems Contact Illumina Technical Support to report any issues with the Pipeline. When reporting an issue, it is critical to capture all the output and error messages produced by a run. This is done by redirecting the output using nohup or the facilities of a cluster management system. For an explanation of nohup, see Running a GOAT Image Analysis on page 27. It helps to attach the makefile corresponding to the part of the Pipeline that is causing the problem. If there are GERALD-related issues, it helps to post the config.txt file found in the GERALD output folder. For problems relating to specific tiles or files, it is useful to send the output of wc -l and ls -l on these files. Catalog # SY Part # Rev. A

21 Technical Assistance 7 Technical Assistance For technical assistance, contact Illumina Customer Support. Table 1 Contact Illumina Customer Support Contacts Number Toll-free Customer Hotline ILMN ( ) International Customer Hotline ILMN ( ) Illumina Website techsupport@illumina.com MSDSs Material safety data sheets (MSDSs) are available on the Illumina website at Product Documentation If you require additional product documentation, you can obtain PDFs from the Illumina website. Go to When you click on a link, you will be asked to log in to icom. After you log in, you can view or save the PDF. If you do not already have an icom account, then click New User on the icom login screen and fill in your contact information. Indicate whether you wish to receive the icommunity newsletter (a quarterly newsletter with articles about, by, and for the Illumina Community), illuminotes (a monthly newsletter that provides important product updates), and announcements about upcoming user meetings. After you submit your registration information, an Illumina representative will create your account and login instructions to you. Frequently Asked Questions Frequently asked questions are available online. Go to and click on Software, then on Genome Analyzer Pipeline Software. Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide

22 8 CHAPTER 1 Overview Catalog # SY Part # Rev. A

23 Chapter 2 Core Pipeline Concepts Topics 10 Introduction 10 Analysis Modules 13 Understanding the Run Folder 15 Run Folder Structure 15 Images Folder 16 Data Folder 17 Run Folder Naming 18 File Naming 18 Configuration/Parameters 19 Calibration and Input Parameters 19 Quality Scoring 19 Image Offsets 20 Frequency Cross-Talk Matrix 21 Phasing/Prephasing Estimates 21 Sample Information 22 Alignment Algorithms 22 ELAND Algorithm Description Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide 9

24 10 CHAPTER 2 Core Pipeline Concepts Introduction Analysis modules perform the specific tasks of image analysis, base calling, and sequence alignment. During an analysis run, a defined folder structure is generated that captures the output of an instrument run in text files and also contains the configuration files. Configuration files contain calibration and input settings that optimize your analysis run and the alignment programs perform sequence analysis. This chapter describes these core concepts of the Genome Analyzer Pipeline Software. Analysis Modules The Pipeline is divided into the following modules: Firecrest is the module used for image analysis. Firecrest identifies cluster positions, sharpens and enhances clusters through image filtering, removes background noise, detects clusters based on morphological features on the image, and extracts intensities. Bustard is the module used for base calling. Bustard deconvolves the signal from the clusters and applies correction for cross-talk, phasing, and prephasing. Frequency cross-talk The Genome Analyzer uses two lasers and four filters to detect four dyes attached to the four types of nucleotide, respectively. The emission spectra of these four dyes overlap so that the four images are not independent. Pipeline uses a frequency cross-talk matrix to correct for this cross-talk (for more information, see Frequency Cross-Talk Matrix on page 20). Phasing/Prephasing Depending on the efficiency of the fluidics and chemistry of the sequencing reactions, a small number of molecules in each cluster may run ahead of (prephasing) or fall behind (phasing) the current incorporation cycle (see Figure 3). This effect is mitigated by applying corrections during the base calling step (for more information, see Phasing/Prephasing Estimates on page 21). Figure 3 Phasing and Prephasing Generation of Recursive Analyses Linked by Dependency (GERALD) is the module used for sequence alignment and metrics visualization. The following two alignment programs work within the GERALD module: Catalog # SY Part # Rev. A

25 Analysis Modules 11 Efficient Large-Scale Alignment of Nucleotide Databases (ELAND) is very fast and aligns for up to two errors from a reference for the first 32 bases or more bases with ELAND extended. This algorithm is used for any reference larger than 100 Kbases. PhageAlign does an exhaustive alignment (all possible alignments up to arbitrary edit distances), but is slow. Use of Modules The use of these modules depends on the input data the Pipeline starts with (Figure 4): Starting with base calling data generated by SCS real time analysis (RTA), use the GERALD.pl script. Chapter 5, Using GERALD for Sequence Alignment decribes the use of GERALD. Starting with image analysis data generated by SCS real time analysis (or IPAR), use the bustard.py script. After the Bustard module is finished, the bustard.py script calls the subscript for GERALD (GERALD.pl) automatically. Chapter 4, Using Bustard Starting with Base Calling, describes the use of bustard.py. Starting with images from the Genome Analyzer, use the script goat_pipeline.py, named after the General Oligo Analysis Tool (GOAT). The goat_pipeline.py script calls the subscripts for three Pipeline modules: Firecrest, Bustard (bustard.py), and GERALD (GERALD.pl). Chapter 3, Using GOAT Starting with Image Analysis describes the use of GOAT. NOTE When you start with image analysis data, you have to invoke Bustard.The goat_pipeline.py script cannot be used to start with image analysis data in Pipeline v1.3 and up. Top Level Script: GOAT Top Level Script: Bustard Top Level Script: GERALD Figure 4 Pipeline Modules Typically, the analysis begins with the alignment script GERALD.pl, using base calling data generated by SCS real time analysis. However, if you need to reanalyze data, you can start with one of the other scripts and use different parameters. Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide

26 12 CHAPTER 2 Core Pipeline Concepts Running the Pipeline Modules The Pipeline is divided into modules that are managed by the make utility. The make utility is commonly used to build executables from source code and is designed to model dependency trees by specifying dependency rules for files. These dependencies are stored in a file called a makefile. Each Pipeline module is a collection of Perl or Python scripts and C++ executables, and has its own makefile associated with the analysis task. Make has a dual purpose within the Pipeline software: To build executables from source code To perform data analysis steps using the software A run of the Pipeline is a two-stage process: 1. Generate the folders and makefiles using one of the above scripts. 2. Start the Pipeline analysis by executing make. The process is described for the different wokflows in Chapter 3, Using GOAT Starting with Image Analysis Chapter 4, Using Bustard Starting with Base Calling, and Chapter 5, Using GERALD for Sequence Alignment. Catalog # SY Part # Rev. A

27 Understanding the Run Folder 13 Understanding the Run Folder The Pipeline operates in a specific directory called the Run Folder where the images and analysis output files are saved by default in a consistent hierarchical structure. A Run Folder containing SCS real time analysis data is very similar to a Run Folder containing IPAR analysis data, or a Run Folder containing only Pipeline analysis data. Figure 5 illustrates a typical Run Folder after SCS image analysis and base calling, and Pipeline alignment. Figure 5 SCS Real Time Analysis Run Folder Directory Structure Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide

28 14 CHAPTER 2 Core Pipeline Concepts Figure 6 illustrates a typical Run Folder after IPAR image analysis and Pipeline base calling and alignment, or one containing only Pipeline analysis data. Figure 6 IPAR/Pipeline Run Folder Directory Structure The standardized structure, file naming conventions, and file formats of the Run Folder allow for the following: A single point of data storage, logging, and analysis output during and after a run. Catalog # SY Part # Rev. A

29 Understanding the Run Folder 15 Encoding sufficient information to trace the history of the data in the Run Folder back to the laboratory notebook without confusion between instruments, experiments, or sites. Standardized input and output enabling component software to operate flawlessly, regardless of the instrument generating the data. Capturing and encoding enough information to independently reanalyze the data at any time, in such a way that existing extractions of sequence and related data are preserved, and parameters used during any point of the extraction process are captured and related to the subsequent output data. Subsequent analyses to be stored in the Run Folder. The software tools and other user software to implement and enforce these structures and standards. Run Folder Structure Images Folder The Run Folder contains the Images folder and Data folder as illustrated in Figure 5 and Figure 6 above. The Data folder contains Image Analysis folders and the Image Analysis folders contain Basecall folders which contain Sequence folders. The Data folder is created by the Genome Analyzer when a run starts. Any analysis performed on the data, including SCS real time analysis and IPAR analysis, is saved within the Data folder. The Images folder holds the images from every tile for all cycles of sequencing. The Images folder will not be present if only analysis data, not the images, are copied to the analysis server after SCS real time analysis or IPAR analysis. There is an option to send images to a second networked run folder apart from the main/default network destination. Each run of the main Pipeline analysis modules creates a subdirectory in the Data folder of the Run Folder as follows (see Figure 5 and Figure 6 above): Each run of the Pipeline image analysis software (Firecrest) creates a new image analysis output folder in the Data folder. Each run of the Pipeline base calling software (Bustard) creates a new subdirectory in the image analysis subdirectory on which the base calls are based, resulting in a tree-like structure of analyses. Parameters and versions for any given analysis run are logged in the folder structure to make it possible to reconstruct any previous analysis run. You can do multiple analyses of the data using different analysis parameters and the results will not be overwritten. The default naming convention for folders generated by the Pipeline consists of the number of cycles run, the version of the software used for the operation (Firecrest, Bustard), the date the analysis initiated, and the login of the user. If the user initiates a second analysis on the same day, a new folder structure is created and the results from the previous analysis are not overwritten. The Images folder contains a subfolder for each lane that has been sequenced. The folders are named using the following convention where the lane number is padded to three digits: L<lane number> Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide

30 16 CHAPTER 2 Core Pipeline Concepts For example, L001 contains the images taken in the first lane. Each lane folder contains a subfolder for each cycle of sequencing. Each image-cycle subfolder contains four images for every tile, one for each of the four bases. The Image folder naming follows the naming convention C<cycle number>.<iteration number>. Cycle number is indexed and represents the nth cycle. Within each image-cycle subfolder are four tif files for each tile. These files are named using the following convention: <sample>_<lane>_<tile>_<base>.tif In the example, s_1_67_g.tif, the s is the default sample-id. NOTE Sample-IDs must not contain any underscores. Underscores are used as separators between the different identifiers of the filename to allow easy splitting by any software reading these filenames. Data Folder The Data folder contains a hierarchical structure that consists of the image analysis output folder, then the base calling output folder, and then the sequence alignment output folder. A new subfolder is generated each time a set of images is processed by the image analysis module (Firecrest), IPAR, or SCS real time analysis. The data are kept in one file per tile for raw intensities and one file per tile for cluster noise. Firecrest and IPAR use the extension _int.txt and use the extension _nse.txt. SCS real time analysis reports image analysis results in the binary.cif format (intensities) and.cnf format (noise). The Data folder contains one config.xml file in each image analysis folder generated as a result of analyzing sets of images.the config.xml file explicitly records which cycle-image folders were used to generate the raw intensities and noise files, and any parameters used. For a detailed description of the parameters file, see Configuration/Parameters on page 18. Image Analysis Folders The image analysis folders have the following naming structure: The image analysis folder generated by SCS real time analysis is called Intensities The image analysis folder generated by IPAR is called IPAR_1.3 Each image analysis folder generated by Firecrest is named using the following convention: C<first cycle>-<last-cycle>_<analysis module><analysis moduleversion>_<date>_<user> For example, C1-27_Firecrest1.8.20_ _myuser.2 contains the second version of an analysis of cycles 1 27 performed using version of the Firecrest analysis module, run by the user myuser on the 31st of July Catalog # SY Part # Rev. A

31 Understanding the Run Folder 17 Base Calling Folders Each image analysis folder may hold multiple sequence folders with the output of different runs of a base caller package. The base calling folders have the following naming structure: The base calling folder generated by SCS real time analysis is called BaseCalls. Each base calling folder generated by Bustard is named using the following convention: <analysis module><analysis moduleversion>_<date>_<user>[.<version-number>] For example, the folder name Bustard1.8.8_ _myuser.3 represents the third run of the Bustard base caller on 8th of November 2005 by the user myuser. Each base calling folder also holds a config.xml that records any relevant information about the run of the base caller module. Run Folder Naming The top level Run Folder name is generated using three fields to identify the <ExperimentName>, separated by underscores. For example, YYMMDD_machinename_NNNN. You should not deviate from the Run Folder naming convention, as this may cause Pipeline to stop. 1. The first field is a six-digit number specifying the date of the run. The YYMMDD ordering ensures that a numerical sort of Run Folders places the names in chronological order. 2. The second field specifies the name of the sequencing machine. It may consist of any combination of upper or lower case letters, digits, or hyphens, but may not contain any other characters (especially not an underscore). It is assumed that the sequencing instrument is synonymous with the PC controlling it, and that the names assigned to the instruments are unique across the sequencing facility. 3. The third field is a four-digit counter specifying the experiment ID on that instrument. Each instrument should be capable of supplying a series of consecutively numbered experiment IDs (incremental unique index) from the onboard sample tracking database or a LIMS. NOTE It is desirable to keep Experiment-IDs (or Sample-ID) and instrument names unique within any given enterprise. You should establish a convention under which each machine is able to allocate Run Folder names independently of other machines to avoid naming conflicts. A Run Folder named _instrument1_0147 indicates experiment number 147, run on instrument 1, on the 8th of Jan While the date and instrument name specify a unique Run Folder for any number of instruments, the addition of an experiment ID ensures both uniqueness and the ability to relate the contents of the Run Folder back to a laboratory notebook or LIMS. Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide

32 18 CHAPTER 2 Core Pipeline Concepts Additional information is captured in the Run Folder name in fields separated by an underscore from the first three fields. For example, you may want to capture the flow cell number in the Run Folder name as follows: YYMMDD_machinename_XXXX_FCYYY. NOTE When publishing the data to a public database, it is desirable to extend the exclusivity globally, for instance by prefixing each machine with the identity of the sequencing center. File Naming The Pipeline uses the following format for file naming: <sample>_<lane>_[<tile>_][<cycle>_][<id>_]<type>.<filesuffix> Some files are split on a read basis, leading to the file naming: <sample>_<lane>_[read_][<tile>_][<cycle>_][<id>_]<type>.<filesuffix> When a given file type is split on a read basis, the read always appears in the name, even for single-read analysis. Table 2 File Naming Components Component <sample> <lane> Description Alphanumeric string (always s ) Single-digit number identifying a flow cell lane <read> Single-digit number identifying the read (starts at 1) <tile> <cycle> <id> <type> <filesuffix> Four-digit number identifying a tile location in a flow cell lane Two- or three-digit number identifying a sequencing cycle Single-digit number to distinguish files; for example, the different reads of a paired-end read Alphabetical string identifying the type of content stored in the file Suffix to identify the traditional file type Example: s_5_1_0030_qseq.txt is a valid filename. Exception: for image (.tif) files, the <tile> location can have less than four digits. Configuration/ Parameters The Data Folder and subfolders, and the top level Image folder can all contain a configuration file (config.xml), and the top level Run Folder a related.params file. This is intended to contain any parameter data specific to the given level of information held in the folder. For an example of the parameters file, see Configuration/Parameters File Format on page 132. Catalog # SY Part # Rev. A

33 Calibration and Input Parameters 19 Calibration and Input Parameters For an optimal analysis run, the Pipeline needs a number of calibration and input parameters. By default, the Pipeline auto-generates these parameters for each analysis. For samples with biased base compositions, as encountered in many tagbased (for example, Digital Gene Expression) or microrna applications, auto-calibration does not provide perfect results. For such samples, you need to dedicate one lane of the flow cell to a control sample and use the -- control-lane command option to generate analysis parameters. For a detailed description, see Command Line Options for GOAT on page 29. Quality Scoring Image Offsets Base quality value calibration now uses a pre-determined calibration table in Bustard, supplied with the software. Custom calibration (lane autocalibration, calibration using a control lane or specification of an external calibration table) is still supported but not generally recommended and, in particular, lane auto-calibration is no longer the default (see Filtering Parameters on page 85). Since Pipeline release 1.3, the quality scoring scheme is the Phred scoring scheme, encoded as an ASCII character by adding 64 to the Phred value. A Phred score of a base is: Q phred =-10 log 10 (e) where e is the estimated probability of a base being wrong. There are small pixel offsets among the four differently colored images taken of each tile. These are due to slightly different optical paths for the four images collected from each tile. The Pipeline uses a file to correct for this, and also corrects for linear rescaling of the image. Each analysis run creates a file called default_offsets.txt in the Data subfolder of the current Run Folder. The default_offsets.txt file is used for subsequent analysis of the same run. Another default_offsets.txt is located in Instruments/<instrument>, which values will be updated during the first run only. The default_offsets.txt file contains four lines, corresponding to A, C, G, and T respectively, with six values each, using the A image as a reference. The following is an example of a typical default_offsets.txt file: The first two columns in a row correspond to the translational offset of X and Y of the four images (in pixels). Since channel A is the reference (first line), the offsets for A are zero. The slightly different optical paths for the four images collected from each tile result in slightly different scales of the images. This is corrected in the next two columns, which indicate scale factors applied to the image. Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide

34 20 CHAPTER 2 Core Pipeline Concepts A scale factor of 0 indicates that the image does not need to be rescaled. A scale factor of for a 1000 x1000 pixel image indicates that images taken in the corresponding frequency channel tend to be one pixel larger than the reference channel. The last two values are set to zero. Frequency Cross- Talk Matrix The Genome Analyzer uses two different lasers to excite the dye attached to each nucleotide. The emission spectra of these four dyes overlap, so the four images are not independent. As in Sanger sequencing, the frequency crosstalk has to be deconvolved using a frequency cross-talk matrix. The frequency cross-talk is estimated during the base calling run and captured in a file called s_matrix.txt. The s_matrix.txt file is located in the base calling folder as shown in Figure 7. Figure 7 Frequency Cross-Talk Matrix and Phasing File Locations The following is an example of a typical s_matrix.txt file: Catalog # SY Part # Rev. A

35 Calibration and Input Parameters 21 The lines starting with a greater than symbol ( > ) specify the order of the rows and columns in terms of the bases they represent. The matrix elements show how the C, A, T, and G dyes/nucleotides (columns) cross-talk into the C, A, T, and G channels. A normal matrix should be diagonally dominant (diagonal elements tend to be the largest values) with the exception of the top-left and bottom-right corners (A/C and G/T crosstalk respectively). These are not as well-separated due to the fact that both corresponding dyes are excited by the same laser. Phasing/Prephasing Estimates Sample Information Depending on the efficiency of the fluidics and the sequencing reactions, a small number of molecules in each cluster may run ahead (prephasing) or fall behind (phasing) the current incorporation cycle. This effect can be mitigated by applying corrections during the base calling step. The phasing estimates are produced before a run of the base caller module and captured in a file called phasing.xml. The phasing.xml file is located in the Phasing folder as shown in Figure 7. As the estimation uses statistical averaging over many clusters and sequences to estimate the correlation of signal between different cycles, the phasing estimates tend to be more accurate for tiles with larger numbers of clusters and a mixture of different sequences. Samples containing only a small number of different sequences do not produce reliable estimates. Depending on the application, a reference genome may be supplied for the read sequences to be aligned against. Genome Analyzer Pipeline v1.5 and CASAVA v1.0 Software User Guide