The LCG Distributed Database Infrastructure

Transcription

1 The LCG Distributed Database Infrastructure Dirk Düllmann, CERN & LCG 3D DESY Computing Seminar 21. May 07 CERN - IT Department CH-1211 Genève 23 Switzerland

2 Outline of the Talk Why databases and why distributed? LCG Distributed Database Project (LCG 3D) Building blocks for a scalable database services database clusters Distribution techniques streams replication and distributed caching Database and replication monitoring optimisation Experiment models and schedules Status and next steps

3 Configurations Create rec condb data set Online Master Data Storage Pixel Pixel Strips Strips ECAL ECAL HCAL HCAL RPC RPC DT ES CSC Trigger DAQ DCS Conditions DT ES CSC Trigger DAQ DCS EMDB DDD Logbook LHC data Offline Reconstruction Conditions DB ONline subset Formatting Calibration Conditions CERN Computer Center Production Validation Development Configuration Conditions Master copy Offline Reconstruction Conditions DB OFfline subset Tier 0 3

4 Distributed Deployment of Databases (=3D) LCG provided initially an infrastructure for distributed access to file based data and file replication Physics applications (and grid services) depend crucially on relational databases for meta data and require similar services for databases Physics applications and grid services use RDBMS eg configuration, conditions, calibration, event tags, file and collection catalogues, production/transfer workflow LCG sites have already experience in providing RDBMS services Goals for common database project as part of LCG increase the availability and scalability of LCG and experiment database components allow applications to access databases in a consistent, location independent way connect database services via data replication mechanisms shared deployment and administration of this infrastructure during 24 x 7 operation Scope set by LCG PEB: Databases Online, Offline and at LCG Tier sites 4

5 LCG 3D Service Architecture Oracle Streams http cache (Squid) Cross DB copy and MySQL/SQLight files T0 autonomous reliable service O S M S O Online DB autonomous reliable service O F T1 DB backbone all data replicated reliable service T2 local DB cache subset data only local service O database (DB) cluster O S M S M MySQL/SQLight DB file S F Squid FroNTier Read-only access at Tier-1/2 (at least initially) 5

6 Database Services for Physics at CERN in 2002

7 Database Services for Physics at CERN in x7 service based on Oracle 9i / Solaris

8 Database Services for Physics at CERN in x7 service based on Oracle 9i / Solaris Most experiment databases hosted on 2-node (4 CPU) cluster on Solaris with 6GB of RAM 18 disks, 32GB each Veritas Volume Manager

9 Database Services for Physics at CERN in x7 service based on Oracle 9i / Solaris Most experiment databases hosted on 2-node (4 CPU) cluster on Solaris with 6GB of RAM 18 disks, 32GB each Veritas Volume Manager But also increasing many Linux disk servers for LHC and non- LHC data (COMPASS, HARP) Locally attached disk Spread all over the CERN computing centre Several different versions of Linux and Oracle

10 How to build a highly available and scalable DB service?

11 How to build a highly available and scalable DB service? Scaling Up

12 How to build a highly available and scalable DB service? Scaling Up

13 How to build a highly available and scalable DB service? Scaling Up Scaling out -> clustering Storage Area Network

14 How to build a highly available and scalable DB service? Scaling Up Scaling out -> clustering Storage Area Network

15 Chosen Architecture: Database Cluster on Linux CERN LAN Oracle Real Application Cluster (RAC) on commodity hardware Redundancy at all levels (CPU, storage, networking) Oracle 10gR2 and RedHat ES 4 Oracle ASM as volume manager RAC1 lxs5033 GB ethernet switch lxs5037 lxs5038 SANbox 5200 A 2 GB fibre channel switch Infortrend storage lxs5030

16 Hardware configuration Four servers CPU : Intel Pentium-D GHz Memory 2G (ECC) Local Disk S-ATA2 80G 7200 rpm Fiber Channel LSI 7102XP-LC, PCI X 1 RAC group for 3D RAC group for other LCG services SAN Switch : Silkworm ports Backend Raid subsystem: StorageTek B280 Each RAC group shares 1.7TB exported from SAN Dual channel & redundant controller ASGC Hurng-Chun 9 - ASGC

17 Beginning of 2007

20 Beginning of 2007 ~220 Database CPUs ~440 GB of RAM (shared DB cache) ~1100 disks Some 15 DB clusters deployed One production cluster per LHC experiment for offline applications ATLAS Online, COMPASS cluster Number of nodes varying from 4 to 8 Several validation and test clusters 1 or 2 per experiment (typically 2 node clusters) Some hardware allocated for internal use/tests

21 Application Validation and Optimisation Database clusters can amplify effects of poor application design - need scalability test during application release cycle Development DB service Validation DB service Production DB service Significant fraction of DB administrator work needs close collaboration with application developers Focus on application with large resource consumption: File transfer systems (FTS, PhEDEx) Grid catalogs (LFC) Experiment Dashboards Condition data (COOL) Event collections (TAGS)

22 Evolving Database Hardware Need to continuously replace database h/w with next generation CPU and storage still maintaining as few s/w configurations as possible: only a single Linux and Oracle version CPU side - continued performance increase via multi-core Recently tested dual quad-core CPU & 16 GB of RAM performance similar to 5 node RAC built with the hardware currently used Multi core works well for database servers but implies a move to 64-bit Linux & Oracle May run into memory bandwidth limitations with many cores Storage side - slower performance increase Sizing for IO operations per second rather than just volume increase -> disk numbers increase Investigating higher performance disks (eg raptor) and other storage technologies (solid state disks)

23 Database Replication and Distributed Caching CERN - IT Department CH-1211 Genève 23 Switzerland

24 FroNTier Launchpad software Squid caching proxy Load shared with Round-Robin DNS Configured in accelerator mode Peer-to-peer caching Wide open frontier * Tomcat - standard FroNTier servlet Distributed as war file Unpack in Tomcat webapps dir Change 2 files if name is different One xml file describes DB connection Squid Squid Squid Tomcat Tomcat Tomcat FroNTier servlet Round-Robin DNS server1 server2 server3 FroNTier servlet FroNTier servlet *In the past, we required the registration so we could add IP/mask DB to our Access Control List (ACL) at CERN. Recently decided to run in wide-open mode so installations can be tested w/o registration. 13 Sept, 2006 CMS Frontier Report

25 How to keep Databases up-to-date? Asynchronous Replication via Streams capture propagation apply CERN FNAL CNAF CERN IN2P3 RAL Sinica BNL Slide : Eva Dafonte Perez LCG 3D Status Dirk Duellmann 15

26 How to keep Databases up-to-date? Asynchronous Replication via Streams insert into emp values ( 03, Joan,.) CERN capture propagation apply FNAL CNAF CERN IN2P3 RAL Sinica BNL Slide : Eva Dafonte Perez LCG 3D Status Dirk Duellmann 15

28 How to keep Databases up-to-date? Asynchronous Replication via Streams insert into emp values ( 03, Joan,.) CERN capture propagation apply FNAL LCR CNAF CERN IN2P3 RAL Sinica BNL Slide : Eva Dafonte Perez LCG 3D Status Dirk Duellmann 15

29 How to keep Databases up-to-date? Asynchronous Replication via Streams insert into emp values ( 03, Joan,.) CERN capture propagation apply FNAL LCR CNAF CERN IN2P3 RAL Sinica BNL Slide : Eva Dafonte Perez LCG 3D Status Dirk Duellmann 15

30 How to keep Databases up-to-date? Asynchronous Replication via Streams insert into emp values ( 03, Joan,.) CERN capture propagation apply FNAL LCR LCR LCR CNAF LCR LCR CERN IN2P3 LCR LCR LCR RAL Sinica BNL Slide : Eva Dafonte Perez LCG 3D Status Dirk Duellmann 15

31 How to keep Databases up-to-date? Asynchronous Replication via Streams insert into emp values ( 03, Joan,.) CERN capture propagation apply FNAL LCR LCR LCR CNAF LCR LCR CERN IN2P3 LCR LCR LCR RAL Sinica BNL Slide : Eva Dafonte Perez LCG 3D Status Dirk Duellmann 15

33 Database s/w Licenses and Support Tier 1 licenses acquired Experiment and grid service request collected for all T1 sites License payment and signed agreement forms received LCG Support ID has been created with Oracle Accounts for Oracle support (MetaLink) created for all T1 database contacts access to Oracle s/w, patches, security upgrades and problem database allows to file problem reports directly to Oracle

34 Tier 0 setup and operational procedures Main streams operations have been included in the Tier 0 DBA operations manual Automated alerts for database or streams problems from 3D monitoring Integrated with GGUS (grid user support) so far: handling of streams problems during working hours only Downstream capture setup has been installed as part of the planned Tier 0 service extension log-mining step runs on a separate box, offloading the source database same h/w and s/w setup as db server nodes

35 Further Decoupling between Databases Objectives Remove impact of capture from Tier 0 database Isolate destination sites from each other pair capture process + queue x each target site big Streams pool size redundant events ( x number of queues) capture process propagation jobs COPY redo log files capture process CERN CERN RAC capture process FNAL CNAF SOURCE DATABASE DOWNSTREAM CERN DESTINATION SITES Slide : Eva Dafonte Perez LCG 3D Status Dirk Duellmann 18

36 Database Backup / Recovery with Streams Collected DB / streams recovery scenarios Recovery after T1 data loss - OK RAL recovered and re-synchronised Replication CENR to CNAF continued unaffected Recovery after T0 data loss - OK Coordinated point-in-time recovery - OK Service procedure documented and validated with two Tier 1 sites and CERN Full recovery exercise with all sites scheduled for the 3D Workshop at CNAF June Recover Tier 1 database from tape Resynchronise replication streams While T0 database is being populated

37 Streams Performance Tuning Focus recently: Wide Area Network Remote sites significantly affected by latency eg ASCG with 300ms round trip time Studied TCP level data flow between CERN and Taiwan Resulting Optimisations increased TCP buffer size (OS level) decrease frequency of acknowledgements between src and dest DB Total improvement of factor 10 Now: 4000 logical change records / sec Checklist for Tier 1 sites has been prepared Focus moved now to LAN setup optimisation

38 Experiment Database Activities & Plans CERN - IT Department CH-1211 Genève 23 Switzerland

39 LHCb COOL service model (Marco Clemencic, 3D workshop 13 Sep 2006) GRIDKA RAL IN2P3 CNAF SARA PIC COOL (Oracle) Two servers at CERN essentially for online and offline Replication to Tier1 s from the online database is a two-step replication Online server at the pit (managed by LHCb); offline server in the CC Andrea Valassi Conditions Databases Rimini, 7 May 2007

40 LFC Replication Testbed Read Only Clients Population Clients Population Clients LFC R-W Server LFC R-W Server LFC Read-Only Server lxb0717.cern.c h rls1r1.cern.ch lxb0716.cern.c h lfc-streams.cr.cnaf.infn.it LFC Oracle Server Master DB Oracle Streams WAN LFC Oracle Server lfc-replica.cr.cnaf.infn.it Replica DB

41 File Catalogue replication between CERN and CNAF LFC replication via streams between CERN and CNAF in production since last November Requested by LHCb to provide read-only catalog replicas Stable operation without major problems Several site interventions at CNAF have been performed Site restart and resynchronisation worked Rate is low compared to conditions In contact with LHCb for adding remaining LHCb Tier 1 sites

42 ATLAS COOL service model COOL Oracle services at Tier0 and ten Tier1 s Two COOL servers at CERN for online/offline (similar to LHCb) Online database within the Atlas pit network, but physically in the CC In addition: Oracle at three muon calibration center Tier2 s ATLAS pit Computer centre Outside world Online / PVSS / HLT farm ATLAS pit network (ATCN) Dedicated 10Gbit link Online OracleDB gateway Streams replication CERN public network Offline master CondDB Calibration updates Tier-0 Sqlite replica (1 file/run) Tier-1 replica Tier-1 replica Tier-0 GRIDKA TAIWAN RAL IN2P3 CNAF SARA BNL TRIUMF PIC Nordugrid (Sasha Vaniachine and Richard Hawkings, 3D workshop 14 Sep 2006) Andrea Valassi Conditions Databases Rimini, 7 May 2007

43 ATLAS Muon Calibration Data Flow Picture: H. von der Schmitt

44 3D Database Resource Request and Current Predictions Request Conditions Challenges (April - Jul) Predictions Dual CPU DB DB Storage Nodes [TB usable] no change wrt GDB Nov ATLAS: COOL + TAGs LHCb: COOL+ LFC r/o replica next review after initial CDC phase (eg May) Dress Rehearsals ATLAS: 4GB on 64bit DB server (Jul-Nov) LHCb: 2 LFC r/o servers in place expect resource upgrade: double storage and CPU review LHC Startup 3+x 1.0 ATLAS (from Nov) 2+y 0.3 LHCb from June ATLAS COOL + TAGs(tbc by ATLAS) nominal year LCG 3D Project Status - 4

45 CMS conditions data at CERN ORCON and ORCOFF conditions data are in POOL_ORA format ORCON-ORCOFF Oracle Streams prototype set up (integration RAC). Production set up later in Vincenzo Innocente, CMS Software Week, April 2007 Andrea Valassi Conditions Databases Rimini, 7 May 2007

46 Slide: L. Lueking FroNTier Launchpad Setup T0 Farm WAN CERN DNS round Robbin Provides load balancing and failover 3 servers running Frontier & Squid (worker nodes) Backend Oracle Database 10gR2 (4-node RAC) 11 Oct

47 CMS Request Update No major Change until August Rough Estimates of CMS DB Resources March 2007 through August 2007 Area Disk Concurrent Users Transactions (CMS contact) Maximum (GB) Peak usage Peak usage (Hz) Online P Offline Conditions 500 (DB) 10 (DB)* 10 (DB)* Tier-0 (CMSR) 100 (per Squid) 10 (Squid) 10 (Squid) 2-3 Squids * Incl. On2off Xfer * Incl. On2off Xfer Offline Conditions Tier-1 (each site) 100 (per Squid) 2-3 Squids/site 10(per Squid) >100! all sites 10 per (Squid) >100! all sites Offline DBS Tier-0 (CMSR) (currently ~2) (currently ~5) 26 Jan 2007 CMS Req. Update and Plans 9 LCG 3D Project Status - 5

48 Deployment Status and Next Steps CMS tested Frontier/SQUID setup during CSA 06 now some 30 Tier 1 and Tier 2 sites connected ATLAS and LHCb moved production mode in April All ten Tier 1 database sites integrated into a single distributed database infrastructure ASGC, BNL, CNAF, GridKA, IN2P3, SARA/ NIKHEF, NDGF, PIC, RAL, TRIUMF One of the largest distributed database setups worldwide Preparing for Tier 1 replica scaling test with O(100) client nodes using ATLAS off line framework ATHENA In summer: participate in WLCG Dress Rehearsal tests

49 More information at WLCG 3D Project or LCG3DWiki CERN Physics Database Service WLCG Persistency Framework