RHIC Computing Facility

Transcription

1 RHIC Cmputing Facility Bruce G. Gibbard DOE/Nuclear Physics Review f RHIC Science and Technlgy BNL July 2005

2 2 RHIC Cmputing Facility (RCF) Organizatinally established in 1997 Staffed as a grup in Physics Dept. Equipment in raised flr cmputer area f IT Divisin, Bldg 515 Currently c-lcated and c-perated with ATLAS Cmputing Facility (ACF) at BNL US ATLAS Tier 1 Reginal Center Lng term, 2008+, facilities will be f cmparable scale Currently: ACF capacities are ~25% f RCF ACF staff level ~ 65% f RCF DOE/Nuclear Physics Review f RHIC Science and Technlgy

3 Missin Scale RCF Missin and Scale Online recrding f Raw data Prductin recnstructin f Raw data Primary facility fr data selectin (mining) and analysis Lng term archiving and serving f all data but nt riginally intended fr Mnte Carl generatin Authrized staff f 20 FTE s Histrically ~$2M/year equipment refresh funding (20-25% annual replacement) Current year austerity limited t $1.3M Addressing bslescence Resulting in imprtant cllateral capacity grwth Grwth beynd riginally planned scale will require an increase in such refresh funding in the ut years 3 DOE/Nuclear Physics Review f RHIC Science and Technlgy

4 Experiment / RCF Interactin 4 Weekly Liaisn (Operatins) Meeting Review recent perfrmance and prblems Plan fr near term peratins Experiments / RCF Annual Series f Meetings t Develp Capital Spending Plan Estimate scale f need fr current/cming run Details f distributin f equipment t be prcured Mst recent in early this Spring fr FY 06 funds Peridic Tpical Meetings Fr example: ~Annual Linux farm OS upgrade planning Other User Interactins Web site Truble ticket system (in huse) 3 instances fr RHIC services (1188 tickets fr last year) ATLAS services (649 tickets fr last year) Grid services (114 tickets fr last year) New ticket system (RT pen surce) in trial use DOE/Nuclear Physics Review f RHIC Science and Technlgy

5 Cmputing Requirements Estimate 5 In Respnse t Recmmendatin frm Last Year s Review A Cmprehensive Lng Range Estimate Dne by PHENIX, RCF & STAR in Fall/Winter 2005 Cnclusins published as part f Mid-Term Strategic Plan: Fr the Relativistic Heavy In Cllider, February 14, 2006 Input is Raw Data Vlume fr Each Species & Experiment by Year Mdel fr Requirements Prjectin Assume facility resurce needs scale with raw data vlume Using 2005 Cu-Cu running as a reference With adjustable parameters reflecting expected relative Richness f data set (density f interesting events) Maturity f prcessing sftware Number f recnstructin passes fr each experiment, species, and year DOE/Nuclear Physics Review f RHIC Science and Technlgy

6 6 Cmputing Cst Estimate Requirements Mdel Output Used as Input t Cst Estimate Csting Mdel is based n Recent prcurements Histric trends (Mre's Law and similar technlgy based trends) Assumed migratin frm reliance n centralized SAN RAID 5 disk tward use f Intel/Linux prcessr farm distributed disk fr bulk f strage As will be discussed later Assume use f bvius technlgy evlutin (multi-cre prcessrs, etc.) Fr running scenaris cnsidered, capacity grwth assciated with equipment refresh meets increased capacity requirements in 2007 but increase equipment funding is required in 2008 and beynd Required capacities by year and a funding prfile allwing them t be achieved are shwn n fllwing slide DOE/Nuclear Physics Review f RHIC Science and Technlgy

7 Requirements Estimate fr a Particular Running Scenari 7 FY '06 FY '07 FY '08 FY '09 FY '10 FY '11 Annual Requirement Real Data Vlume (TB) Rec CPU (KSI2K) Analys CPU (KSI2K) Dist. Disk (TB) Cent. Disk (TB) Annual Tape Vlume (TB) Tape bandwidth (MB/sec) WAN bandwidth (Mb/sec) Simulatin CPU (KSI2K) Simulatin Data Vlume (TB) Installed Requirement CPU (KSI2K) Dist. Disk (TB) Cent. Disk (TB) Tape Vlume (TB) Tape bandwidth (MB/sec) WAN bandwidth (Mb/sec) Funding Prfile ($K) FY '06 FY '07 FY '08 FY '09 FY '10 FY '11 CPU + Distributed Disk Central Disk Tape Strage System LAN Overhead Ttal Annual Cst DOE/Nuclear Physics Review f RHIC Science and Technlgy

8 8 Principal RCF Services General Cllabratin & User Supprt Cmpute Services (Linux Farms) Prgrammatic prductin prcessing Individual analysis Online Strage (Disk) Wrking data strage (Read / Write) Data serving (mstly Read) Mass Strage (HSM Rbtic Tape System) Raw data recrding and archiving Derived data archiving Grid & Netwrk Services DOE/Nuclear Physics Review f RHIC Science and Technlgy

9 RCF Staff 9 Current authrized staff level f 20 FTE s is lwer than staffs at ther sites with cmparable missins It des wrk in terms f peratins in the cntext f a c-lcated ATLAS Tier 1 Very high level f cmmnality and therefre imprtant synergy A dramatic divergence in technical directins culd change this but seems very unlikely It des nt allw fr aggressive invlvement in develping and/r adpting new technlgies Which might in the lng r even medium term significantly imprve cst effectiveness Distributin f RHIC Effrt Current Target FTE's FTE's Linux Farms Mass Strage Disk User Supprt Fabric Infrastr Wide Area Services Admin TOTAL DOE/Nuclear Physics Review f RHIC Science and Technlgy

10 10 Mass Strage Cmpute Farm Online Strage DOE/Nuclear Physics Review f RHIC Science and Technlgy

11 11 Cmpute Services Intel/Linux dual CPU rack munted systems Currently 1,700 cmpute servers with 3,400 CPU s (Intel) fr 2,700 ksi2k Delivery is expected in the next week r tw f 170 additinal machines with 800 ksi2k First lcal prductin use f AMD (in place f Intel) and dual cre prcessrs (s 680 effective CPU s in 170 servers) Dual cre AMD Optern is ptimal fr price perfrmance Multi-cre als addresses pwer/cling barrier by finessing nnlinearity f pwer cnsumptin with clck speed Transitin t AMD and t dual cre is transparent / nn-disruptive Expect t address future requirements by cntinuing t fllw Mre s law price/perfrmance in cmmdity market (including multi-cre, & 64 bit advances) DOE/Nuclear Physics Review f RHIC Science and Technlgy

12 12 Grwth f RCF Linux Farm # f CPU 's Year DOE/Nuclear Physics Review f RHIC Science and Technlgy

13 13 Grwth in RCF Cmputatinal Pwer kspecint2k Year DOE/Nuclear Physics Review f RHIC Science and Technlgy

14 Grwth in RCF Cmputatinal Pwer by experiment 14 kspecint2k BRAHMS kspecint2k PHENIX Year Year kspecint2k PHOBOS kspecint2k STAR Year Year DOE/Nuclear Physics Review f RHIC Science and Technlgy

15 Resurce Utilizatin Issues 15 In Respnse t Recmmendatin frm Last Year s Review Ganglia Mnitring by Lad r CPU shw significant time with n apparent active jb in a large fractin f the farm But Cndr mnitring shws that in general there are active jbs Issue is ne f nt prperly accunted I/O waits In particular netwrk I/O s (dcache, NFS write, pssibly thers) But nt in general HPSS DOE/Nuclear Physics Review f RHIC Science and Technlgy

16 Resurce Sharing Queue 16 Gal is t make available idle cycles in prcessr farms t ther experiment users withut impacting primary wner Mechanism is t evict guest user jbs when wner jbs starts System needs further ptimizatin Wrks well fr shrt jbs but nt widely used See case f lng PHENIX jbs versus shrt STAR jbs abve Its use needs t be mre transparent (shrt jbs default t it) Cnsider increasing rights f guests Only suspend fr wner r Give guest a grace perid t cmplete DOE/Nuclear Physics Review f RHIC Science and Technlgy

17 17 Online (Disk) Strage Changes in disk strage mdel As discussed last year, histric RCF mdel was Sun/NFS served RAID 5 SAN cnnected central disk fr all strage augmented by lcal scratch disk n CPU bxes Grwth in demand drve disk csts t match and exceed CPU csts New strategy: Differentiate disk by functin Central NFS full functin disk fr Read/Write nt yet backed up data, f which there is a limited amunt Less expensive distributed disk fr mstly Read already backed up data, f which there is a great deal Cmmitting t supprt tw nline (disk) strage paths DOE/Nuclear Physics Review f RHIC Science and Technlgy

18 18 Full Functin Disk Service Read/Write, reliable, high perfrmance and high availability NFS served RAID systems Histrically ~150 TB f SUN served RAID 5 disk ~70 TB f Panasas (appliance) served RAID 5 disk Current acquisitins ~100 TB f Nexsan & Aberdeen Linux served RAID 5/6 disk Mved t lwer Tier f RAID disk vendrs this year Hped fr very high reliability/perfrmance frm smewhat mre expensive vendrs nt fully realized and ffset by ther technical prblems DOE/Nuclear Physics Review f RHIC Science and Technlgy

19 19 Mstly Read Disk Disk deplyed n Linux prcessr farm ndes IDE & SCSI SATA ~3 x less expensive then full functin disk Requires additinal disk/data management sftware and yields less than a truly full functin slutin Tw such management systems currently in use at RCF dcache - DESY/Fermilab develped Grid aware prduct Scalable, rbust, unified pl f strage with integral mass stre backing, psix-like access, ROOT supprt ATLAS is majr BNL user with TB Heavily used in prductin fr ~18 mnths Xrtd SLAC, CERN, BNL + ther cmmunity develpers Similar scalable prduct currently under very active develpment STAR is majr BNL user with TB Heavily used fr mre than a year DOE/Nuclear Physics Review f RHIC Science and Technlgy

20 Grwth in RCF Distributed Strage Capacity 20 Strage Capacity (TB) Year DOE/Nuclear Physics Review f RHIC Science and Technlgy

21 Sme Detail 21 PHENIX is RHIC dcache user Currently 26 TB f disk n 120 ndes Read rates n ccasin exceed 50 TB/day Expansin t 110 TB expected sn with further expansin t 200 TB likely this year Lcal dcache develpment wrk in cnjunctin with US ATLAS Tier 1 Currently supprting ne f the largest dcache instances Officially accepted as a develper by primary prduct develpers Actively investigating Issues f high perfrmance scalability as a disk system Issues f Grid perfrmance and reliability (in particular its SRM) STAR and xrtd Develpment cycle Accurate usage mnitring (at user applicatin level) Integrate SRM (Strage Resurce Manager) technlgy n the back-end Prceed with develpment t make Xrtd next-generatin Grid-aware Alter mdel t an bject based mdel Built-in capability t scan large datasets in search fr rare event Plan strngly relies n SRM technlgy - a US-leading activity t date Additinal supprt beynd existing ~ 2 FTE x 3 years DOE/Nuclear Physics Review f RHIC Science and Technlgy

22 PHENIX dcache Cnfiguratin 22 HPSS Tape Sil PHENIX Cmpute Farm (107 pl servers, SL3) External Write Pls (9 pl servers, RHEL[3,4]) Cre Server (PNFS, Admin, etc.) Per nde: VA 800Mhz, 400GB disk, Fast Ethernet, EXT and XFS file system 4 SRM/GFTP Drs with fast ethernet 3 dcap Drs with fast Ethernet 26 TB disk in ttal All Pl Servers are als Clients (Wrkers) Outbund Only irewall DOE/Nuclear Physics Review f RHIC Science and Technlgy

23 Xrtd architecture 23 Crdinatin with MSS required integrating with the DataCarusel tl. Redirectr culd be set as rund rbin DOE/Nuclear Physics Review f RHIC Science and Technlgy

24 Mass Strage System 24 Hierarchical Strage Manager is HPSS frm IBM Versin 5.1 mving sn t 6.2 StrageTek Rbtic Tape Libraries 4 classic cylindrical sils 1 new SL8500 linear library Ttal capacity ~7 PBytes nw cntaining ~4 PBytes f data Tape Drives 37 StrageTek 9940b (30 MB/sec) 20 LT0 Gen 3 (80 MB/sec) ~30 TB f disk cache New data mvers are Linux/Intel instead f AIX/IBM In huse develped tape access ptimizatin sftware DOE/Nuclear Physics Review f RHIC Science and Technlgy

25 25 Histry f Raw Data Cllectin Raw Data Cllected in RHIC Runs 400 Terabytes BRAHMS PHENIX PHOBOS STAR Experiment Run 2 Au-Au Run 2 p-p Run 3 d-au Run 3 p-p Run 4 Au-Au 200 Run 4 Au-Au 63 Run 4 p-p Run 5 Cu-Cu 100 Run 5 Cu-CU 62 Run 5 p-p Run 6 p-p DOE/Nuclear Physics Review f RHIC Science and Technlgy

26 26 Tape I/O Rate (GB/day) ver Past Year Vlume f Data (TB) Stred in HPSS ver Six Years DOE/Nuclear Physics Review f RHIC Science and Technlgy

27 Grid and Netwrk Services 27 Cmputing mdels f RHIC experiments predate the Grid S, unlike ATLAS, they were nt riginally based n a Grid cmputing mdel Hwever the desire t utilize substantial distributed resurces is driving evlutin f these mdels tward Grid cmputing Originally fr simulatin but nw mre generally fr analysis LBNL, etc. fr STAR Riken, etc. fr PHENIX Same staff engaged in majr US ATLAS Grid effrt als supprts RHIC wide area distributed cmputing with Supprt fr Grid tls and services as well as netwrk expertise GridFTP, SRM, etc High vlume netwrk transfer ptimizatin Supprt fr invlvement (f STAR) in Open Science Grid OSG sftware deplyment and integratin f resurce int OSG OSG administratin: registratin, authrizatin, etc. DOE/Nuclear Physics Review f RHIC Science and Technlgy

28 RHIC Grid Cnfiguratin at BNL 28 Grid Infrastructure HPSS LSF clusters/ Cndr pls HPSS MOVER Grid Users RHIC/STAR Jb scheduler Gatekeepers HRM/GridFTP SERVERS Panasas Submit Grid Jbs Grid Jb Requests GridFTP NFS Disks Internet DOE/Nuclear Physics Review f RHIC Science and Technlgy

29 29 PHENIX Data Transfer t CCJ Curtesy f Y. Watanabe Ttal data transferred t CCJ (Cmputer Center in Japan) was 309 TB f Raw Data Average transfer rate was ~3TB/day! And 6TB/day max speed. RCF supplied Netwrk/Grid expertise includeing: Installatin f Grid transfer and mnitring tls Truble shting and tuning f netwrk path Integratin dcache/srm (Strage Resurce Manager) fr future transfers DOE/Nuclear Physics Review f RHIC Science and Technlgy

30 30 Wide Area Netwrk Jan 06 WAN upgrade f BNL cnnectivity OC48 (2.5 Gb/sec) 2 x λ (20 Gb/sec) Mtivated by heavy netwrk use by bth RHIC and ATLAS (ptential cntentin) Funded in equal part by ESnet, DOE NP, DOE HEP and BNL Cnnectin still lacks desired redundancy and diversity Will require significant additinal funding nt yet identified DOE/Nuclear Physics Review f RHIC Science and Technlgy

31 Physical Infrastructure 31 Last year the limit f capacities n existing flr space were reached fr chilled water, cled air, UPS pwer, and pwer distributin Therefre this year fr first time majr physical infrastructure imprvements were needed 250KW f lcal UPS / PDU systems t supprt new prcurements New chilled water feed Lcal rack tp cling fr new prcurements These are nw installed and peratinal with csts cvered by Lab GPP funds Appraching limit f available flr space itself Raised flr with fire detectin & suppressin, physical security ATLAS expansin, with limited equipment retirement, is driving space grwth (while bth prgrams cntinue pwer/cling grwth) Reallcatin f space t RCF/ACF will allw 2007 and 2008 expansin Additinal pwer & cling will be needed each year Need fr expansin f space itself in 2009 and beynd is fully recgnized by Lab management but detailed plan fr addressing this need is nt yet in place DOE/Nuclear Physics Review f RHIC Science and Technlgy

32 Cyber Security 32 Facility is a firewall prtected enclave within the BNL firewall prtect site Gd wrking relatinship with BNL ITD cyber security Facility has a ~single sign-n Kerbers base authenticatin infrastructure Cyber security lapses f the last year All a result f externally sniffed passwrds (netwrk r keystrke) One incident resulted in a rt level cmprise f a single machine but nne result in the utilizatin f facility resurces against ther sites Majr current effrts Cntributing t new BNL Cyber Security Prgram Plan Deplying, facility wide, Ord (BNL develped hst based cnfiguratin tracking/auditing tl fr Unix-like systems) Preparing fr expect mve t nn-reusable (ne time) passwrds in next tw mnths (addressing sniffed passwrd vulnerability) Cncern remains f ptential future cnflicts between Grid requirements, regulatry requirements, and a cyber security plicy/architecture which des nt disrupt effective facility use DOE/Nuclear Physics Review f RHIC Science and Technlgy

33 Cnclusins 33 Plans t evlve and expand facility services t meet expected needs Are based n s far successful adjustments f technical directins Remain within the mainstream f the HENP cmputing Will require agreed and planned fr increases in 2008 and beynd Physical infrastructure needs are recgnized but lng term plans are nt yet fully in place Specifically the facility will needs new space with apprpriate characteristics and services fr 2009 and beynd Immediate annual increases in pwer/cling capacity and csts are nging and are being addressed The Grid is likely t significantly change future cmputing Based n ATLAS pathfinder experience at BNL, RCF shuld be well psitined t explit its advantages Cyber Security remains a lming presence DOE/Nuclear Physics Review f RHIC Science and Technlgy