EMBL-EBI. 3D databases and data warehouse technology

Transcription

1 3D databases and data warehouse technology

2 Overview Overall Strategy Terms and background Populating the databases Clean up processes How can I use the database? What next

3 What is a database? By the term database we refer to the system rather than the data Indexed file space Also used as a shorthand for a database management system (DBMS) Methods for accessing and changing data Controls for referential integrity

4 Normalisation Data fields in a normalised database appear only once CHAIN ID A attr 185 RESIDUE CHAIN ID A A SEQ COMP ID ASP COMPONENT Data fields in a denormalised database are repeated in different places 1 2 LYS ID ASP attr -1 LYS +1 CHAIN ID attr A 185 RESIDUE CHAIN ID A SEQ 1 COMP ID ASP A 2 LYS CHAINattr COMPattr COMPONENT ID attr ASP -1 LYS +1

5 Structural hierarchy assembly molecule (entity) chain residue

6 ASU and assemblies assembly ASU chain chain residues residues

7 The pipeline archive services edited cif archive DB data warehouse manual edit post-load processes distribution

8 The first steps archive services edited cif archive DB data warehouse manual edit post-load processes distribution

9 The first steps A series of scripts Parses non-standard header records Fills in chain identifiers Outputs a first cut clean file Manual editing ~1000 entries require manual editing The result is a format file that can be passed to the subsequent automatic steps

10 bizarre errors 1ew1 ATOM 47 N6 A A ATOM 59 1H6 A A ATOM 60 2H6 A A ATOM 47 N6 A A ATOM 59 1H6 A A ATOM 60 2H6 A A

11 automatic processing archive services edited cif archive DB data warehouse manual edit post-load processes distribution

12 process details Automatic cleanup (d2c) Incorporates quaternary structure information Runs a lot of checks and corrections Outputs mmcif file Loading Metadata-driven custom loader Load through views with insert triggers Many heuristics also applied to data within these triggers

13 Using reference data Variations in legacy data Hinders accurate searches Hinders links to other services Match data against controlled vocabularies Within scripts Within database during load Semi-automated Use string matching algorithms Effective when controlled vocabulary well maintained $COLI COLI E. COLI E.COLI ESCHERCHIA COLI ESCHERICHI $COLI ESCHERICHIA $ COLI ESCHERICHIA $COLI ESCHERICHIA COLI ESCHERICHIA COLI. EXCHERICHIA COLI EXPRESCHERICHIA COLI

14 Chemical Components More difficult to deal with Where coordinates and nomenclature do not agree, have to make a judgement on which, if either, are correct We maintain a curated database of compounds, against which legacy data is compared atom nomenclature ongoing; relatively easy to correct where the compound has been correctly identified Stereochemistry may indicate that the compound name is incorrect

15 Ligand nomenclature Ligands are often named inconsistently or even entirely incorrectly, e.g. -Dmannose (MAN) vs. -D-mannose (BMA) MAN Errors are detected using a graphbased structure comparison algorithm BMA

16 not all cases resolvable 1d7t DTY 4 in chain A, model 1 - is it D or L?? HEADER DE NOVO PROTEIN 19-OCT-99 1D7T TITLE NMR STRUCTURE OF AN ENGINEERED CONTRYPHAN CYCLIC PEPTIDE TITLE 2 (MOTIF CPXXPXC) MODRES 1D7T DTY A 4 TYR D-TYROSINE HET DTY A 4 21 HETNAM DTY D-TYROSINE FORMUL 1 DTY C9 H11 N1 O3

17 post-load processing archive services edited cif archive DB data warehouse manual edit post-load processes distribution

18 process details Involved in deriving data and building crosslinks to other services Geometric information Analysing non-polymer components and assembling full entities from individual components Links to taxonomy and sequence databases

19 transformation to DW archive services edited cif archive DB data warehouse manual edit post-load processes distribution

20 process details Set of SQL scripts Supports Oracle (routinely) and MySQL (development) Periodically undertake full transform takes a couple of weeks Provide weekly incremental patches much faster Supports transforms into different data marts

21 coming soon Continuing cleanup HET group curation Sequence cross-references Citations More choice on downloads Data marts (even single tables) Groups of entries Release of clean files (end 2006)

22 who did what archive services edited cif archive DB data warehouse manual edit post-load processes distribution