DNA Sequencing Data Compression. Michael Chung
|
|
|
- Dayna Ball
- 8 years ago
- Views:
Transcription
1 DNA Sequencing Data Compression Michael Chung
2 Problem DNA sequencing per dollar is increasing faster than storage capacity per dollar. Stein (2010)
3 Data 3 billion base pairs in human genome Genomes are 99.9% similar 95% SNPs
4 Data
5 Data
6 Data
7 Data
8 Data
9 Data
10 Data
11 Data
12 Data
13 Data
14 Data
15 Data
16 Compression Tools Fixed Codes VINT Golomb / Golomb-Rice Elias Codes MOV Codes Variable Codes Huffman Codes
17 Variable Integers (VINT) Standard integer data type uses 4 bytes Example: Binary code for 5:
18 Variable Integers (VINT) Standard integer data type uses 4 bytes Example: Binary code for 5: Small integers don t need full 4 bytes Last bit of each byte used as a flag: 0: VINT continues to next byte 1: End of VINT
19 Golomb / Golomb-Rice Encode an integer with two bit strings Preamble Mantissa Choose parameter m to encode integer j:
20 Golomb / Golomb-Rice Example Encode 61 with m = 8:
21 Delta Encoding Store relative integers, not absolute integers Instead of 2, 4, 6, 8, 10 Store 2, 2, 2, 2, 2
22 Elias Codes Binary code for 5: Can we compress the leading zeroes?
23 Elias Codes To encode integer j:
24 Monotone Value Coding (MOV Coding) Encode j 1, j 2,, j k Encode j 1 with Elias Gamma encoding Preamble: (log j i log j i-1 ) 0-bits Mantissa: binary code for j i starting with most significant 1 bit
25 Monotone Value Coding (MOV Coding) 2: : 11 4: : 101 6: 110 7: 111 8:
26 Huffman Encoding I will not pass notes in class. I will not pass notes in class. I will not pass notes in class. I will not pass notes in class. I will not pass notes in class.
27 Huffman Encoding Let 0 = I will not pass notes in class
28 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
29 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
30 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
31 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
32 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
33 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
34 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
35 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
36 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
37 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
38 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
39 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
40 Huffman Encoding Huffman encoding assigns shorter tags to more frequently encountered data.
41 Papers 2008 Human genomes as attachments Scott Christley, Yiming Lu, Chen Li, Xiaohui Xie 2010 Data structures and compression algorithms for high-throughput sequencing technologies Kenny Daily, Paul Rigor, Scott Christley, Xiaohui Xie, Pierre Baldi 2011 Efficient Storage of high throughput DNA sequencing data using reference-based compression Markus His-Yang Fritz, Rasko Leinonen, Guy Cochrane, Ewan Birney
42 Papers 2008 Human genomes as attachments Scott Christley, Yiming Lu, Chen Li, Xiaohui Xie 2010 Data structures and compression algorithms for high-throughput sequencing technologies Kenny Daily, Paul Rigor, Scott Christley, Xiaohui Xie, Pierre Baldi 2011 Efficient Storage of high throughput DNA sequencing data using reference-based compression Markus His-Yang Fritz, Rasko Leinonen, Guy Cochrane, Ewan Birney
43 Summary
44 Summary Genomes are 99.9% similar Store only the differences
45 Assumptions
46 Method
47 Method
48 Techniques SNP Mapping
49 Techniques SNP Mapping NCBI dbsnp 11 million SNPs 99% are bi-allele
50 Techniques VINT and Delta Positions Use for indels (insertions and deletions)
51 Techniques VINT and Delta Positions Use for indels (insertions and deletions)
52 Techniques VINT and Delta Positions Use for indels (insertions and deletions) Deletion Example
53 Techniques K-mer partitioning
54 Techniques K-mer partitioning
55 Techniques K-mer partitioning
56 Techniques K-mer partitioning
57 Results James Watson s Genome 3.5 M differences 3.3 M SNPs 220 K indels
58 Review Good general concepts used in later research Compare to reference genome Take advantage of: Common SNPs Position integer compression + delta encoding Frequency analysis based compression (Huffman)
59 Papers 2008 Human genomes as attachments Scott Christley, Yiming Lu, Chen Li, Xiaohui Xie 2010 Data structures and compression algorithms for high-throughput sequencing technologies Kenny Daily, Paul Rigor, Scott Christley, Xiaohui Xie, Pierre Baldi 2011 Efficient Storage of high throughput DNA sequencing data using reference-based compression Markus His-Yang Fritz, Rasko Leinonen, Guy Cochrane, Ewan Birney
60 Summary Compress position and length integers with various compression methods Fixed code compression works well
61 Assumptions
62 Method Substitution triples
63 Method Substitution triples
64 Method Substitution triples
65 Method Substitution triples
66 Results Compressed Data Size (Bytes) Compression Method Dataset 1 Dataset 2 Dataset 3 GenCompress 56,166,419 36,099, ,541,330 gzip 41,378,624 95,688, ,818,824 bzip2 42,233,336 94,030, ,061,616 7zip 30,651,664 83,319, ,811,520
67 Results Relative Elias Gamma (REG) indexed encoding performs best Huffman coding can outperform when dataset is highly repetitive
68 Review Pros Efficient encoding of integers Can handle read-level compression Cons Storage space used to represent data identical to reference Does not handle quality scores Assumes aligned data
69 Papers 2008 Human genomes as attachments Scott Christley, Yiming Lu, Chen Li, Xiaohui Xie 2010 Data structures and compression algorithms for high-throughput sequencing technologies Kenny Daily, Paul Rigor, Scott Christley, Xiaohui Xie, Pierre Baldi 2011 Efficient Storage of high throughput DNA sequencing data using reference-based compression Markus His-Yang Fritz, Rasko Leinonen, Guy Cochrane, Ewan Birney
70 Summary Lossy compression using quality budget
71 Summary Lossy compression using quality budget Include extra sequences in reference common unmapped reads bacterial / viral sequences
72 Summary Lossy compression using quality budget Include extra sequences in reference common unmapped reads bacterial / viral sequences Support paired-end reads
73 Summary Lossy compression using quality budget Include extra sequences in reference common unmapped reads bacterial / viral sequences Support paired-end reads bits / base pair
74 Assumptions
75 Method Relative Golomb encoding for positions Elias Gamma encoding for deletion lengths Huffman encoding for read lengths, quality scores
76 Method Quality score budget
77 Method Quality score budget
78 Results Bits per base Compression Method NA12878 chrom20 Pseudomonas syringae pathovar syringae B728a Bzip BAM Reference-based, 2% quality budget Bzip2, referencebased, 2% quality budget
79 Results bits / base pair Compared to bzip2 (1 bit / base pair) 2 hours for compression
80 Review Pros Stores quality scores Handles some unaligned data Allows for controlled loss of precision Can handle read-level compression Cons Storage space used to represent data identical to reference Still cannot efficiently store all unaligned data
81 What is too expensive to compress? Image data? Quality scores? Unaligned reads?
82 Conclusions No single compression method is optimal for all data sets Choose assumptions about data carefully Must make decisions about which data to discard, depending on goals Need a centralized database of genomic sequence data Short Read Archive (SRA) Things to consider Storage Speed Accessibility Annotation and mass data organization Scalable, reliable, secure distributed system design Business
83 References 2008 Human genomes as attachments Scott Christley, Yiming Lu, Chen Li, Xiaohui Xie 2010 Data structures and compression algorithms for high-throughput sequencing technologies Kenny Daily, Paul Rigor, Scott Christley, Xiaohui Xie, Pierre Baldi 2010 The case for cloud computing in genome informatics Lincoln D. Stein 2011 Efficient Storage of high throughput DNA sequencing data using reference-based compression Markus His-Yang Fritz, Rasko Leinonen, Guy Cochrane, Ewan Birney
Efficient storage of high throughput DNA sequencing data using reference-based compression
Method Efficient storage of high throughput DNA sequencing data using reference-based compression Markus Hsi-Yang Fritz, Rasko Leinonen, Guy Cochrane, and Ewan Birney 1 European Molecular Biology Laboratory
Efficient storage of high throughput DNA sequencing data using reference-based compression
Efficient storage of high throughput DNA sequencing data using reference-based compression Markus Hsi-Yang Fritz, Rasko Leinonen, Guy Cochrane and Ewan Birney EMBL-EBI, Wellcome Trust Genome Campus, Hinxton,
DNA Mapping/Alignment. Team: I Thought You GNU? Lars Olsen, Venkata Aditya Kovuri, Nick Merowsky
DNA Mapping/Alignment Team: I Thought You GNU? Lars Olsen, Venkata Aditya Kovuri, Nick Merowsky Overview Summary Research Paper 1 Research Paper 2 Research Paper 3 Current Progress Software Designs to
Arithmetic Coding: Introduction
Data Compression Arithmetic coding Arithmetic Coding: Introduction Allows using fractional parts of bits!! Used in PPM, JPEG/MPEG (as option), Bzip More time costly than Huffman, but integer implementation
LZ77. Example 2.10: Let T = badadadabaab and assume d max and l max are large. phrase b a d adadab aa b
LZ77 The original LZ77 algorithm works as follows: A phrase T j starting at a position i is encoded as a triple of the form distance, length, symbol. A triple d, l, s means that: T j = T [i...i + l] =
Tutorial for Windows and Macintosh. Preparing Your Data for NGS Alignment
Tutorial for Windows and Macintosh Preparing Your Data for NGS Alignment 2015 Gene Codes Corporation Gene Codes Corporation 775 Technology Drive, Ann Arbor, MI 48108 USA 1.800.497.4939 (USA) 1.734.769.7249
Accelerating Data-Intensive Genome Analysis in the Cloud
Accelerating Data-Intensive Genome Analysis in the Cloud Nabeel M Mohamed Heshan Lin Wu-chun Feng Department of Computer Science Virginia Tech Blacksburg, VA 24060 {nabeel, hlin2, wfeng}@vt.edu Abstract
Next Generation Sequencing: Technology, Mapping, and Analysis
Next Generation Sequencing: Technology, Mapping, and Analysis Gary Benson Computer Science, Biology, Bioinformatics Boston University gbenson@bu.edu http://tandem.bu.edu/ The Human Genome Project took
Removing Sequential Bottlenecks in Analysis of Next-Generation Sequencing Data
Removing Sequential Bottlenecks in Analysis of Next-Generation Sequencing Data Yi Wang, Gagan Agrawal, Gulcin Ozer and Kun Huang The Ohio State University HiCOMB 2014 May 19 th, Phoenix, Arizona 1 Outline
Towards Integrating the Detection of Genetic Variants into an In-Memory Database
Towards Integrating the Detection of Genetic Variants into an 2nd International Workshop on Big Data in Bioinformatics and Healthcare Oct 27, 2014 Motivation Genome Data Analysis Process DNA Sample Base
Compression techniques
Compression techniques David Bařina February 22, 2013 David Bařina Compression techniques February 22, 2013 1 / 37 Contents 1 Terminology 2 Simple techniques 3 Entropy coding 4 Dictionary methods 5 Conclusion
SRA File Formats Guide
SRA File Formats Guide Version 1.1 10 Mar 2010 National Center for Biotechnology Information National Library of Medicine EMBL European Bioinformatics Institute DNA Databank of Japan 1 Contents SRA File
Challenges associated with analysis and storage of NGS data
Challenges associated with analysis and storage of NGS data Gabriella Rustici Research and training coordinator Functional Genomics Group gabry@ebi.ac.uk Next-generation sequencing Next-generation sequencing
Information, Entropy, and Coding
Chapter 8 Information, Entropy, and Coding 8. The Need for Data Compression To motivate the material in this chapter, we first consider various data sources and some estimates for the amount of data associated
On the Use of Compression Algorithms for Network Traffic Classification
On the Use of for Network Traffic Classification Christian CALLEGARI Department of Information Ingeneering University of Pisa 23 September 2008 COST-TMA Meeting Samos, Greece Outline Outline 1 Introduction
A Tutorial in Genetic Sequence Classification Tools and Techniques
A Tutorial in Genetic Sequence Classification Tools and Techniques Jake Drew Data Mining CSE 8331 Southern Methodist University jakemdrew@gmail.com www.jakemdrew.com Sequence Characters IUPAC nucleotide
Analysis of Compression Algorithms for Program Data
Analysis of Compression Algorithms for Program Data Matthew Simpson, Clemson University with Dr. Rajeev Barua and Surupa Biswas, University of Maryland 12 August 3 Abstract Insufficient available memory
Image Compression through DCT and Huffman Coding Technique
International Journal of Current Engineering and Technology E-ISSN 2277 4106, P-ISSN 2347 5161 2015 INPRESSCO, All Rights Reserved Available at http://inpressco.com/category/ijcet Research Article Rahul
Version 5.0 Release Notes
Version 5.0 Release Notes 2011 Gene Codes Corporation Gene Codes Corporation 775 Technology Drive, Ann Arbor, MI 48108 USA 1.800.497.4939 (USA) +1.734.769.7249 (elsewhere) +1.734.769.7074 (fax) www.genecodes.com
How-To: SNP and INDEL detection
How-To: SNP and INDEL detection April 23, 2014 Lumenogix NGS SNP and INDEL detection Mutation Analysis Identifying known, and discovering novel genomic mutations, has been one of the most popular applications
CHAPTER 2 LITERATURE REVIEW
11 CHAPTER 2 LITERATURE REVIEW 2.1 INTRODUCTION Image compression is mainly used to reduce storage space, transmission time and bandwidth requirements. In the subsequent sections of this chapter, general
Complete Genomics Sequencing
TECHNOLOGY OVERVIEW Complete Genomics Sequencing Introduction With advances in nanotechnology, high-throughput instruments, and large-scale computing, it has become possible to sequence a complete human
SeqScape Software Version 2.5 Comprehensive Analysis Solution for Resequencing Applications
Product Bulletin Sequencing Software SeqScape Software Version 2.5 Comprehensive Analysis Solution for Resequencing Applications Comprehensive reference sequence handling Helps interpret the role of each
Focusing on results not data comprehensive data analysis for targeted next generation sequencing
Focusing on results not data comprehensive data analysis for targeted next generation sequencing Daniel Swan, Jolyon Holdstock, Angela Matchan, Richard Stark, John Shovelton, Duarte Mohla and Simon Hughes
Abstract. Results. Introduction. Overview
1 Compression of FASTQ and SAM Format Sequencing Data James K. Bonfield 1,, Matthew V. Mahoney 2 1 Wellcome Trust Sanger Institute, Cambridge, UK 2 Dell, Inc., USA E-mail: jkb@sanger.ac.uk Abstract Storage
A Catalogue of the Steiner Triple Systems of Order 19
A Catalogue of the Steiner Triple Systems of Order 19 Petteri Kaski 1, Patric R. J. Östergård 2, Olli Pottonen 2, and Lasse Kiviluoto 3 1 Helsinki Institute for Information Technology HIIT University of
LifeScope Genomic Analysis Software 2.5
USER GUIDE LifeScope Genomic Analysis Software 2.5 Graphical User Interface DATA ANALYSIS METHODS AND INTERPRETATION Publication Part Number 4471877 Rev. A Revision Date November 2011 For Research Use
Wan Accelerators: Optimizing Network Traffic with Compression. Bartosz Agas, Marvin Germar & Christopher Tran
Wan Accelerators: Optimizing Network Traffic with Compression Bartosz Agas, Marvin Germar & Christopher Tran Introduction A WAN accelerator is an appliance that can maximize the services of a point-to-point(ptp)
Compression of next-generation sequencing reads aided by highly efficient de novo assembly
Nucleic Acids Research Advance Access published August 16, 2012 Nucleic Acids Research, 2012, 1 9 doi:10.1093/nar/gks754 Compression of next-generation sequencing reads aided by highly efficient de novo
RETRIEVING SEQUENCE INFORMATION. Nucleotide sequence databases. Database search. Sequence alignment and comparison
RETRIEVING SEQUENCE INFORMATION Nucleotide sequence databases Database search Sequence alignment and comparison Biological sequence databases Originally just a storage place for sequences. Currently the
Inverted Indexes: Trading Precision for Efficiency
Inverted Indexes: Trading Precision for Efficiency Yufei Tao KAIST April 1, 2013 After compression, an inverted index is often small enough to fit in memory. This benefits query processing because it avoids
Big Data Technology Map-Reduce Motivation: Indexing in Search Engines
Big Data Technology Map-Reduce Motivation: Indexing in Search Engines Edward Bortnikov & Ronny Lempel Yahoo Labs, Haifa Indexing in Search Engines Information Retrieval s two main stages: Indexing process
Trends in Genome Compression
Trends in Genome Compression Sebastian Wandelt, Marc Bux, and Ulf Leser Knowledge Management in Bioinformatics, Institute for Computer Science, Humboldt-Universität zu Berlin, Germany June 4, 2013 Abstract
Analysis of NGS Data
Analysis of NGS Data Introduction and Basics Folie: 1 Overview of Analysis Workflow Images Basecalling Sequences denovo - Sequencing Assembly Annotation Resequencing Alignments Comparison to reference
Copy Number Variation: available tools
Copy Number Variation: available tools Jeroen F. J. Laros Leiden Genome Technology Center Department of Human Genetics Center for Human and Clinical Genetics Introduction A literature review of available
Next generation sequencing (NGS)
Next generation sequencing (NGS) Vijayachitra Modhukur BIIT modhukur@ut.ee 1 Bioinformatics course 11/13/12 Sequencing 2 Bioinformatics course 11/13/12 Microarrays vs NGS Sequences do not need to be known
Multimedia Systems WS 2010/2011
Multimedia Systems WS 2010/2011 31.01.2011 M. Rahamatullah Khondoker (Room # 36/410 ) University of Kaiserslautern Department of Computer Science Integrated Communication Systems ICSY http://www.icsy.de
Data formats and file conversions
Building Excellence in Genomics and Computational Bioscience s Richard Leggett (TGAC) John Walshaw (IFR) Common file formats FASTQ FASTA BAM SAM Raw sequence Alignments MSF EMBL UniProt BED WIG Databases
encoding compression encryption
encoding compression encryption ASCII utf-8 utf-16 zip mpeg jpeg AES RSA diffie-hellman Expressing characters... ASCII and Unicode, conventions of how characters are expressed in bits. ASCII (7 bits) -
HENIPAVIRUS ANTIBODY ESCAPE SEQUENCING REPORT
HENIPAVIRUS ANTIBODY ESCAPE SEQUENCING REPORT Kimberly Bishop Lilly 1,2, Truong Luu 1,2, Regina Cer 1,2, and LT Vishwesh Mokashi 1 1 Naval Medical Research Center, NMRC Frederick, 8400 Research Plaza,
Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay
Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 17 Shannon-Fano-Elias Coding and Introduction to Arithmetic Coding
SAP HANA Enabling Genome Analysis
SAP HANA Enabling Genome Analysis Joanna L. Kelley, PhD Postdoctoral Scholar, Stanford University Enakshi Singh, MSc HANA Product Management, SAP Labs LLC Outline Use cases Genomics review Challenges in
Data Reduction: Deduplication and Compression. Danny Harnik IBM Haifa Research Labs
Data Reduction: Deduplication and Compression Danny Harnik IBM Haifa Research Labs Motivation Reducing the amount of data is a desirable goal Data reduction: an attempt to compress the huge amounts of
Parallel Compression and Decompression of DNA Sequence Reads in FASTQ Format
, pp.91-100 http://dx.doi.org/10.14257/ijhit.2014.7.4.09 Parallel Compression and Decompression of DNA Sequence Reads in FASTQ Format Jingjing Zheng 1,* and Ting Wang 1, 2 1,* Parallel Software and Computational
Binary Number System. 16. Binary Numbers. Base 10 digits: 0 1 2 3 4 5 6 7 8 9. Base 2 digits: 0 1
Binary Number System 1 Base 10 digits: 0 1 2 3 4 5 6 7 8 9 Base 2 digits: 0 1 Recall that in base 10, the digits of a number are just coefficients of powers of the base (10): 417 = 4 * 10 2 + 1 * 10 1
17 July 2014 WEB-SERVER MANUAL. Contact: Michael Hackenberg (hackenberg@ugr.es)
WEB-SERVER MANUAL Contact: Michael Hackenberg (hackenberg@ugr.es) 1 1 Introduction srnabench is a free web-server tool and standalone application for processing small- RNA data obtained from next generation
4 5 6 7 8 9 10 11 What is a character acte set? Definition Usage A character encoding or character set (sometimes referred to as code page) consists of a code that pairs a sequence of characters from a
Genome Data Compression using Digital Chaos
Genome Data Compression using Digital Chaos Sai Venkatesh Balasubramanian Sree Sai Vidhya Mandhir, Mallasandra, Bengaluru-560109, Karnataka, India saivenkateshbalasubramanian@gmail.com Abstract: Efficient
Big Data & Scripting Part II Streaming Algorithms
Big Data & Scripting Part II Streaming Algorithms 1, Counting Distinct Elements 2, 3, counting distinct elements problem formalization input: stream of elements o from some universe U e.g. ids from a set
THE SECURITY AND PRIVACY ISSUES OF RFID SYSTEM
THE SECURITY AND PRIVACY ISSUES OF RFID SYSTEM Iuon Chang Lin Department of Management Information Systems, National Chung Hsing University, Taiwan, Department of Photonics and Communication Engineering,
Dimensionality Reduction of DNA Sequences through Wavelet Transforms
Dimensionality Reduction of DNA Sequences through Wavelet Transforms MAMTA C. PADOLE Department of Computer Science and Engineering, The Maharaja Sayajirao University of Baroda Kalabhavan, Baroda, Gujarat,
UCLA Team Sequences Cell Line, Puts Open Source Software Framework into Production
Page 1 of 6 UCLA Team Sequences Cell Line, Puts Open Source Software Framework into Production February 05, 2010 Newsletter: BioInform BioInform - February 5, 2010 By Vivien Marx Scientists at the department
BROADBAND AND HIGH SPEED NETWORKS
BROADBAND AND HIGH SPEED NETWORKS INTRODUCTION TO MUTIPLEXING Multiplexing is the set of techniques that allows the simultaneous transmission of multiple signals across a single data link INTRODUCTION
Today s topics. Digital Computers. More on binary. Binary Digits (Bits)
Today s topics! Binary Numbers! Brookshear.-.! Slides from Prof. Marti Hearst of UC Berkeley SIMS! Upcoming! Networks Interactive Introduction to Graph Theory http://www.utm.edu/cgi-bin/caldwell/tutor/departments/math/graph/intro
Extensible Sequence (XSQ) File Format Specification 1.0.1
Extensible Sequence (XSQ) File Format Specification 1.0.1 Table of Contents 1 INTRODUCTION... 1 2 FILE FORMAT... 1 3 GENERALIZATIONS AND EXTENDED SPECIFICATION... 11 4 FIGURES... 13 1 Introduction This
Data Storage 3.1. Foundations of Computer Science Cengage Learning
3 Data Storage 3.1 Foundations of Computer Science Cengage Learning Objectives After studying this chapter, the student should be able to: List five different data types used in a computer. Describe how
Lempel-Ziv Coding Adaptive Dictionary Compression Algorithm
Lempel-Ziv Coding Adaptive Dictionary Compression Algorithm 1. LZ77:Sliding Window Lempel-Ziv Algorithm [gzip, pkzip] Encode a string by finding the longest match anywhere within a window of past symbols
Indexing and Compression of Text
Compressing the Digital Library Timothy C. Bell 1, Alistair Moffat 2, and Ian H. Witten 3 1 Department of Computer Science, University of Canterbury, New Zealand, tim@cosc.canterbury.ac.nz 2 Department
8/7/2012. Experimental Design & Intro to NGS Data Analysis. Examples. Agenda. Shoe Example. Breast Cancer Example. Rat Example (Experimental Design)
Experimental Design & Intro to NGS Data Analysis Ryan Peters Field Application Specialist Partek, Incorporated Agenda Experimental Design Examples ANOVA What assays are possible? NGS Analytical Process
Streaming Lossless Data Compression Algorithm (SLDC)
Standard ECMA-321 June 2001 Standardizing Information and Communication Systems Streaming Lossless Data Compression Algorithm (SLDC) Phone: +41 22 849.60.00 - Fax: +41 22 849.60.01 - URL: http://www.ecma.ch
Comparing Methods for Identifying Transcription Factor Target Genes
Comparing Methods for Identifying Transcription Factor Target Genes Alena van Bömmel (R 3.3.73) Matthew Huska (R 3.3.18) Max Planck Institute for Molecular Genetics Folie 1 Transcriptional Regulation TF
Secured Lossless Medical Image Compression Based On Adaptive Binary Optimization
IOSR Journal of Computer Engineering (IOSR-JCE) e-issn: 2278-0661, p- ISSN: 2278-8727Volume 16, Issue 2, Ver. IV (Mar-Apr. 2014), PP 43-47 Secured Lossless Medical Image Compression Based On Adaptive Binary
Go where the biology takes you. Genome Analyzer IIx Genome Analyzer IIe
Go where the biology takes you. Genome Analyzer IIx Genome Analyzer IIe Go where the biology takes you. To published results faster With proven scalability To the forefront of discovery To limitless applications
Class Notes CS 3137. 1 Creating and Using a Huffman Code. Ref: Weiss, page 433
Class Notes CS 3137 1 Creating and Using a Huffman Code. Ref: Weiss, page 433 1. FIXED LENGTH CODES: Codes are used to transmit characters over data links. You are probably aware of the ASCII code, a fixed-length
Deep Sequencing Data Analysis: Challenges and Solutions
29 Deep Sequencing Data Analysis: Challenges and Solutions Ofer Isakov and Noam Shomron Sackler Faculty of Medicine, Tel Aviv University, Israel 1. Introduction Ultra high throughput sequencing, also known
Succinct Data Structures: From Theory to Practice
Succinct Data Structures: From Theory to Practice Simon Gog Computing and Information Systems The University of Melbourne January 5th 24 Big data: Volume, Velocity,... ...and Variety Picture source IBM
Deep Sequencing Data Analysis
Deep Sequencing Data Analysis Ross Whetten Professor Forestry & Environmental Resources Background Who am I, and why am I teaching this topic? I am not an expert in bioinformatics I started as a biologist
Issues in Data Storage and Data Management in Large- Scale Next-Gen Sequencing
Issues in Data Storage and Data Management in Large- Scale Next-Gen Sequencing Matthew Trunnell Manager, Research Computing Broad Institute Overview The Broad Institute Major challenges Current data workflow
Shouguo Gao Ph. D Department of Physics and Comprehensive Diabetes Center
Computational Challenges in Storage, Analysis and Interpretation of Next-Generation Sequencing Data Shouguo Gao Ph. D Department of Physics and Comprehensive Diabetes Center Next Generation Sequencing
zdelta: An Efficient Delta Compression Tool
zdelta: An Efficient Delta Compression Tool Dimitre Trendafilov Nasir Memon Torsten Suel Department of Computer and Information Science Technical Report TR-CIS-2002-02 6/26/2002 zdelta: An Efficient Delta
Data Storage. Chapter 3. Objectives. 3-1 Data Types. Data Inside the Computer. After studying this chapter, students should be able to:
Chapter 3 Data Storage Objectives After studying this chapter, students should be able to: List five different data types used in a computer. Describe how integers are stored in a computer. Describe how
Reference-free compression of high throughput sequencing data with a probabilistic de Bruijn graph
Reference-free compression of high throughput sequencing data with a probabilistic de Bruijn graph Gaëtan Benoit, Claire Lemaitre, Dominique Lavenier, Erwan Drezen, Thibault Dayris, Raluca Uricaru, Guillaume
The Relational Way To Dam The Flood Of Genome Data
The Relational Way To Dam The Flood Of Genome Data Sebastian Dorok supervised by Prof. Gunter Saake Bayer Pharma AG and University of Magdeburg Germany sebastian.dorok@ovgu.de ABSTRACT Mutations in genomes
Global Alliance. Ewan Birney Associate Director EMBL-EBI
Global Alliance Ewan Birney Associate Director EMBL-EBI Our world is changing Research to Medical Research English as language Lightweight legal Identical/similar systems Open data Publications Grant-funding
SMALL INDEX LARGE INDEX (SILT)
Wayne State University ECE 7650: Scalable and Secure Internet Services and Architecture SMALL INDEX LARGE INDEX (SILT) A Memory Efficient High Performance Key Value Store QA REPORT Instructor: Dr. Song
The string of digits 101101 in the binary number system represents the quantity
Data Representation Section 3.1 Data Types Registers contain either data or control information Control information is a bit or group of bits used to specify the sequence of command signals needed for
Data Processing of Nextera Mate Pair Reads on Illumina Sequencing Platforms
Data Processing of Nextera Mate Pair Reads on Illumina Sequencing Platforms Introduction Mate pair sequencing enables the generation of libraries with insert sizes in the range of several kilobases (Kb).
Molecular typing of VTEC: from PFGE to NGS-based phylogeny
Molecular typing of VTEC: from PFGE to NGS-based phylogeny Valeria Michelacci 10th Annual Workshop of the National Reference Laboratories for E. coli in the EU Rome, November 5 th 2015 Molecular typing
HADOOP IN THE LIFE SCIENCES:
White Paper HADOOP IN THE LIFE SCIENCES: An Introduction Abstract This introductory white paper reviews the Apache Hadoop TM technology, its components MapReduce and Hadoop Distributed File System (HDFS)
Probability Interval Partitioning Entropy Codes
SUBMITTED TO IEEE TRANSACTIONS ON INFORMATION THEORY 1 Probability Interval Partitioning Entropy Codes Detlev Marpe, Senior Member, IEEE, Heiko Schwarz, and Thomas Wiegand, Senior Member, IEEE Abstract
Video compression: Performance of available codec software
Video compression: Performance of available codec software Introduction. Digital Video A digital video is a collection of images presented sequentially to produce the effect of continuous motion. It takes
Genotyping by sequencing and data analysis. Ross Whetten North Carolina State University
Genotyping by sequencing and data analysis Ross Whetten North Carolina State University Stein (2010) Genome Biology 11:207 More New Technology on the Horizon Genotyping By Sequencing Timeline 2007 Complexity
Module 1. Sequence Formats and Retrieval. Charles Steward
The Open Door Workshop Module 1 Sequence Formats and Retrieval Charles Steward 1 Aims Acquaint you with different file formats and associated annotations. Introduce different nucleotide and protein databases.
Work Package 13.5: Authors: Paul Flicek and Ilkka Lappalainen. 1. Introduction
Work Package 13.5: Report summarising the technical feasibility of the European Genotype Archive to collect, store, and use genotype data stored in European biobanks in a manner that complies with all
Raima Database Manager Version 14.0 In-memory Database Engine
+ Raima Database Manager Version 14.0 In-memory Database Engine By Jeffrey R. Parsons, Senior Engineer January 2016 Abstract Raima Database Manager (RDM) v14.0 contains an all new data storage engine optimized
Using Illumina BaseSpace Apps to Analyze RNA Sequencing Data
Using Illumina BaseSpace Apps to Analyze RNA Sequencing Data The Illumina TopHat Alignment and Cufflinks Assembly and Differential Expression apps make RNA data analysis accessible to any user, regardless
HIV NOMOGRAM USING BIG DATA ANALYTICS
HIV NOMOGRAM USING BIG DATA ANALYTICS S.Avudaiselvi and P.Tamizhchelvi Student Of Ayya Nadar Janaki Ammal College (Sivakasi) Head Of The Department Of Computer Science, Ayya Nadar Janaki Ammal College
Package PKI. July 28, 2015
Version 0.1-3 Package PKI July 28, 2015 Title Public Key Infrastucture for R Based on the X.509 Standard Author Maintainer Depends R (>= 2.9.0),
When you install Mascot, it includes a copy of the Swiss-Prot protein database. However, it is almost certain that you and your colleagues will want
1 When you install Mascot, it includes a copy of the Swiss-Prot protein database. However, it is almost certain that you and your colleagues will want to search other databases as well. There are very
Integrating NLTK with the Hadoop Map Reduce Framework 433-460 Human Language Technology Project
Integrating NLTK with the Hadoop Map Reduce Framework 433-460 Human Language Technology Project Paul Bone pbone@csse.unimelb.edu.au June 2008 Contents 1 Introduction 1 2 Method 2 2.1 Hadoop and Python.........................
Modified Golomb-Rice Codes for Lossless Compression of Medical Images
Modified Golomb-Rice Codes for Lossless Compression of Medical Images Roman Starosolski (1), Władysław Skarbek (2) (1) Silesian University of Technology (2) Warsaw University of Technology Abstract Lossless
Lossless Compression of Cloud-Cover Forecasts for Low-Overhead Distribution in Solar-Harvesting Sensor Networks
Lossless Compression of Cloud-Cover Forecasts for Low-Overhead Distribution in Solar-Harvesting Sensor Networks Christian Renner and Phu Anh Tuan Nguyen ENSsys 14, Memphis, TN, USA November 6 th, 2014
Practical Guideline for Whole Genome Sequencing
Practical Guideline for Whole Genome Sequencing Disclosure Kwangsik Nho Assistant Professor Center for Neuroimaging Department of Radiology and Imaging Sciences Center for Computational Biology and Bioinformatics
Importance of Statistics in creating high dimensional data
Importance of Statistics in creating high dimensional data Hemant K. Tiwari, PhD Section on Statistical Genetics Department of Biostatistics University of Alabama at Birmingham History of Genomic Data
SDGen: Mimicking Datasets for Content Generation in Storage Benchmarks
SDGen: Mimicking Datasets for Content Generation in Storage Benchmarks Raúl Gracia-Tinedo (Universitat Rovira i Virgili, Spain) Danny Harnik, Dalit Naor, Dmitry Sotnikov (IBM Research-Haifa, Israel) Sivan
Binary Numbering Systems
Binary Numbering Systems April 1997, ver. 1 Application Note 83 Introduction Binary numbering systems are used in virtually all digital systems, including digital signal processing (DSP), networking, and
Assuring the Quality of Next-Generation Sequencing in Clinical Laboratory Practice. Supplementary Guidelines
Assuring the Quality of Next-Generation Sequencing in Clinical Laboratory Practice Next-generation Sequencing: Standardization of Clinical Testing (Nex-StoCT) Workgroup Principles and Guidelines Supplementary
Delivering the power of the world s most successful genomics platform
Delivering the power of the world s most successful genomics platform NextCODE Health is bringing the full power of the world s largest and most successful genomics platform to everyday clinical care NextCODE
Acceleration for Personalized Medicine Big Data Applications
Acceleration for Personalized Medicine Big Data Applications Zaid Al-Ars Computer Engineering (CE) Lab Delft Data Science Delft University of Technology 1" Introduction Definition & relevance Personalized
Compression of next-generation sequencing quality scores using memetic algorithm
PROCEEDINGS Open Access Compression of next-generation sequencing quality scores using memetic algorithm Jiarui Zhou 1,2, Zhen Ji 2*, Zexuan Zhu 2, Shan He 3 From 2013 International Conference on Intelligent