Processing Genome Data using Scalable Database Technology. My Background

Transcription

1 Johann Christoph Freytag, Ph.D. Stanford University, February 2004 Harvard Univ. Visiting Scientist, Microsoft Res. (2002) My Background Professor of ERCR (European Computer Industry Research Centre), München (87-89) DEC s Database Technology Center, München (90-93) Starburst project, IBM Almaden Research Center (85-87) Visiting Scientist, Almaden Research Center (97/98) Visiting Scientist, IBM SVL (2001) nur zum nicht-kommerziellen Gebrauch 1

2 What s the Meaning of Life DNA RNA Protein Genomic Transmitter Transcription? Messenger Translation? Gene product Replication!? Overview (Biological) Motivation/Problems Using Database Technology Gene-EYe Integration-Platform Data Cleansing BLAST-Integration into GDB In-and-Out-the-Database: Using Workflow for dry Experiments Summary nur zum nicht-kommerziellen Gebrauch 2

3 View of Biological Areas Environment Diseases Experiments Pathways Life Evolution DNA Genome RNA Transcriptome Amino Acids Proteome Biological Motivation View of Data Data Source Environment Diseases Experiments OMIM Express Brenda Pathways Life Evolution Gene Ontology Taxonomy KEGG EMBL DNA RefSeq Genome LocusLink RNA EMBL Transcriptome (EST) Amino Acids SWISS-PROT Proteome Interpro Biological Motivation nur zum nicht-kommerziellen Gebrauch 3

4 Complex Relationships A graph depicting the relationships between 400+ biological data sources served by the EBI via SRS Database Growth of EMBL (# of records) More than 400 Data Sources on the WEB Source: DBIS (our) Approach SwissProt EMBL Database Model of the Biological World ESEMBLE... KABAT nur zum nicht-kommerziellen Gebrauch 4

5 (Biological) Motivation/Problems Using Scalable Database Technology Gene-EYe Integration-Platform Data Cleansing BLAST-Integration into GDB In-and-Out-the-Database: Using Workflow Concepts for dry Experiments Summary Overview Gene-EYe Integration-Platform Vision Provide mechanisms for unified handling of different data sources data source integration change management user defined data preparation Provide relevant tools for sequence manipulation and retrieval work flow support for operation and administration Gen-Eye Vision nur zum nicht-kommerziellen Gebrauch 5

6 Gene-EYe Integration-Platform The Big Picture Genome Data Warehouse Layer (GDW Schema) KNOWLEDGE Biological Entities -> Biological Concepts (e.g. Life Cycle) Genome DataBase Layer (GDB Schema) CONTENT Relational Entities -> Biological Entities (e.g. Gene) Genome Data Store Layer (GDS Schema) DATA Flat File Data -> Relational Entities (e.g. EMBL) Design GDS: From Flat File to Database Genome Data Store Layer (GDS Schema) Data Storage Data Cleansing Update/Admin GDS Load Tools GDS Admin Tools ENSEMBL DDL InterPro DDL TAXO DDL SWALL DDL EMBL DDL ENSEMBL scanner InterPro scanner TAXO scanner SWALL scanner EMBL scanner nur zum nicht-kommerziellen Gebrauch 6

7 The Data Import Pipeline - Revisited Data File Scanner Load Files Loader Summary Instance Spec. CLOB Files Load Spec. Gene-EYe GDS Format Spec. Content Spec. DDL-Gen. DDL Script Controller Phase 1: Property Files Perl scripts Hand crafted Phase 2: GEM 1 Repository de.hui.dbis.geneeye.* (Java) Autogenerate from Metadata 1: CWM compliant GeneEYe Metadata Repository Modeling the Maintenance Process nur zum nicht-kommerziellen Gebrauch 7

8 GDB-Layer: From Data to Biology Genome Database Layer (GDB Schema) Data Integration Data Cleansing (Sem.) Queries Data GDB Builder (IBM Clio?) Schema Gene Protein Transcript Tissue Variant [Data] EMBL SWALL TAXO InterPro ENSEMBL GDB Mapper (IBM Clio) [Definition] Defined by and in cooperation w/ domain experts Genome Data Store Layer (GDS Schema) Data Storage Data Cleansing (Syn.) Update/Admin Schema Mapping with Clio with permission of Dr. Felix Naumann IBM Almaden Research Center Clio Source Schema User mapping Clio Target Schema DB SQL or XQuery DB nur zum nicht-kommerziellen Gebrauch 8

9 Clio Features with permission of Dr. Felix Naumann IBM Almaden Research Center Schema Viewer Visual mapping between schema elements Attribute Matcher Intelligent suggestions of likely mappings Data Viewer Data examples for mapping queries Queries SQL, XSLT, Xquery Use and adhere to source and target schema constraints GDW: Providing Facts for Research Genome Data Warehouse Layer (GDW Schema) Data Mining Ontology Mapping Process Simulation Ontology GDW Miner GDB Explorer Variant Tissue Transcript Protein Gene Variant Tissue Transcript Protein Gene Genome Database Layer (GDB Schema) Data Integration Data Cleansing (Sem.) Queries nur zum nicht-kommerziellen Gebrauch 9

10 (Biological) Motivation/Problems Using Scalable Database Technology Gene-EYe Integration-Platform Data Cleansing BLAST-Integration into GDB In-and-Out-the-Database: Using Workflow Concepts for dry Experiments Summary Overview Errors in Genome Data DNA Sequence Determination Classes of errors in genome data production Genome Experimental errors DNA Feature Gene Annotation Analysis errors mrna Transformation errors Propagated errors Stale data Protein Sequence Determination [Müller, Naumann, Freytag, ICIQ, 2003] only 1.3 % difference Protein Function SUBUNIT Annotation The main difference are transcript DISEASE copy numbers in the brain DNA, RNA Sequence agagattagcgcgctagatcgatatgataga 0,23% gctatatcatccgagatagcagatagctcta gcacactattacacgagcagcgaccttatat 2,58% Structure Annotation Protein Sequence MDDREDLVYQAKLAEQAERYDEMVESMKKVD AGMDVELTVEERNLLSVAYKNVIGARRASWY RIISSIEQKEENKGGEDKLKMIREYRQMVER FUNCTION Function Annotation 5% - 30% MAY ACT AS INTRACELLULAR SIGNALING COMPONENT... BINDS DIRECTLY TO 5% ZO-1-40% INVOLVED IN ACUTE LEUKEMIAS nur zum nicht-kommerziellen Gebrauch 10

11 Reliability-based Merging (cont.) Domain expert identifies reliable parts for merging Definition of a set of views for integration Current work: r 1 ID A 1 A 2 A 3 1 A A B r 1 4 B mismatch patterns? 5 C How to Merge assess their r 2 ID A 1 A 2 A 3 1 B B B B D Which are the relevant relevance & importance? ID A 1 A 2 A 3 1 A A B B C e.g. MIN() (Biological) Motivation/Problems Using Scalable Database Technology Gene-EYe Integration-Platform BLAST-Integration into GDB In-and-Out-the-Database: Using Workflow Concepts for dry Experiments Summary Overview nur zum nicht-kommerziellen Gebrauch 11

12 BLAST: General Introduction DC-File Algorithm/Package: Similarity Search Devloped by Altschul et al. (1990) Three Steps: FormatDB Preprocessing 1. Search for Word Pairs (Iseq, DSeq) of Length L on the Data Collection of Sequences above Threshhold T 2. Expansion of each Word Pair until the Value V of their Alignment is away from the local maximum 3. Output of complete alignment (Highscoring Segment Pair, HSP), if Value(Alignment) > S Index Sequences BLAST Report Features Query sequence BLAST Call Output: Powerset of Alignments BLAST UDF Implementation Goal: Using BLAST in SQL-statements How? BLAST-UDF implemented as Table Function Use in SQL Query SELECT * FROM TABLE( BLAST(<Parameter>, <Query Sequence>, <Comparison Sequence> )) Each call returns a set of alignments over Sequences in the Database nur zum nicht-kommerziellen Gebrauch 12

13 Structure of UDB Table Function Implementation: Mapping of program into calling structure for table functions Communiaction between the different calls via scratchpad scratchpad: Storage area which remains intact and unchanged between UDF calls Storage of data structures for different steps especially for output from postprocessing: SeqAlign For all Initialize SEQUENCES Alignment without gaps Postprocessing For all UDF BLAST ALIGNMENTS Output of the results Release data structures related with sequences Release global data structures FIRST OPEN FETCH CLOSE FINAL (Biological) Motivation/Problems Using Scalable Database Technology Gene-EYe Integration-Platform BLAST-Integration into GDB In-and-Out-the-Database: Using Workflow Concepts for dry Experiments Summary Overview nur zum nicht-kommerziellen Gebrauch 13

14 The Challenge: Exon Skipping Gene Protein One Gen with 100 Exons ~ Variations n Exons within one Gene linearly combined (splicing) Used as Pattern for Protein Generation Challenge: Exon Skipping Do alternative fusion points new funktional (i.e. biologigacl meaningful) patterns? nur zum nicht-kommerziellen Gebrauch 14

15 Functional Genomics: Gain of New Insight First Horizon: Simple Exon Skipping New Functionality! Flow of Processing Steps Generate Exon Sequence Local Database (automatic) Remote Tool (Web Based) Find Similarity Search Supported by local DB Check for Biological Validity nur zum nicht-kommerziellen Gebrauch 15

16 Implementation Some facts 60 days 100% load One splice form per minute So far: ca splice forms First biolog. meaningful results Cooperations Cooperation with Univ. of Jena (Rolf Backofen) Berlin Center of Bioinformatics (BCB) Charite, FU, Max-Planck-Institut (M. Vingron) Industry: IBM, small companies, Patrick Chappatte, Switzerland nur zum nicht-kommerziellen Gebrauch 16

17 Database Environment IBM p-server (sponsored by IBM) CPU CPU CPU CPU 2.3 TByte CPU CPU CPU CPU DB2 Summary Lesson learnt Highly Dynamic Environment Data: changes frequently User: changes frequently Provide a framework for Date integration Data processing Data changes Data dependencies.. Meta data management Future Work Query processing Include domain knowledge Data cleansing Set of UDFs for biological data processing Visualization of Data Summary nur zum nicht-kommerziellen Gebrauch 17