Managing managed storage

Transcription

1 Managing managed storage CERN Disk Server operations HEPiX 2004 / BNL Data Services team: Vladimír Bahyl, Hugo Caçote, Charles Curran, Jan van Eldik, David Hughes, Gordon Lee, Tony Osborne, Tim Smith

2 Outline Which are our Data Services? Disk server CERN Management tools What s next? 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 2

3 A lot of hardware Disk storage 350 storage in a box Linux diskservers 6700 disks 550 TeraBytes of raw disk space Tape storage 2 robotic installations each with 5 STK 9310 silos B drives, tapes, 2.8 PB drives, 8000 tapes, 160 TB 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 3

4 Many applications repair/spare 11% AFS 3% CDR 6% Oracle 11% R&D 14% CASTOR 55% 200 CASTOR! 40 Oracle 20 CDR 10 AFS scratch dcache, LCG, OpenLab, EGEE, data challenges 40 in repair/spare A very heterogeneous environment! And very dynamic too 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 4

5 Players Many teams involved: Application responsibles / Users Service managers System administrators team Suppliers Software often not redundant need to minimize downtime! so the hardware should be! 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 5

6 Storage in a box 13 different hardware configurations: 8 26 IDE disks, hot-swappable trays Ware RAID controllers 2 CPUs 2 3 power supplies GigE network card Should be redundant 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 6

7 hardware interventions 55 interventions since Sep 1 disk replacements (70%) trays, cables, fans, PSU 33% involve (un)scheduled downtime Older hardware harder to maintain One supplier out of business Incidents to spice up life 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 7

8 Disk replacement % Broken Mirrors 4.5% 4.0% 3.5% 3.0% 2.5% 2.0% 1.5% 1.0% 0.5% Christmas 1224 disks replaced Jumbo servers 0.0% Dec-03 Jan-04 Feb-04 Mar-04 Apr-04 May-04 Jun-04 Jul-04 Aug-04 Sep months before case agreed: Head instabilities 4 weeks to execute 1224 disks exchanged (=18%); And the cages as well 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 8

9 65 Jumbo s TB raw disk space Ware controllers 600 MHz PIII No PXE Becoming hard to maintain Many still under warranty Make good mini-bars! 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 9

10 175 4U servers 4U (5U) rack mounted TB 2 * 3-Ware 7000 series currently upgrading firmware 2 * 1 GHz PIII s No PXE (yet) Various maintenance issues 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 10

11 115 8U servers 8U rack mounted Tb 3 4 * 3-Ware 7500(6)-8 2 * 2.4 GHz Xeon Well controlled, well maintained, well behaved, after disk replacements 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 11

12 Diskserver evolution Capacity [GB] Gross capacity Usable capacity Price/usable GB CHF/GB 0 Oct-00 Apr-01 Oct-01 Apr-02 Oct-02 Apr-03 0 Tender date 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 12

13 That was then HW RAID1 Ext2 filesystems many of them 13 different kernels! RedHat 6.1/6.2, 7.2/7.3, 2.1ES Need for automation + standardization ELFms toolsuite Quattor installation + configuration LEMON performance + exception monitoring LEAF Hardware and State Management 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 13

14 this is now RedHat 7.3, preparing for SLC3 Oracle: RHEL 2.1, preparing RHEL 3 kernel has old 3-Ware driver HW RAID5 + hot spare disk Up to 50% more usable space On 3-Ware 7000 controller with up-to-date firmware SW RAID0 + XFS Improved performance expected iozone benchmark Old XFS version Improved kernel / elevator tuning 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 14

15 Updating the toolbox SMART to predict disk failure daily and weekly self-tests, on every disk IPMI v1.5 HW monitoring and event control Power control, resets Lm_sensors temperature monitoring Hardware and software specific All data flows into Lemon repository 19-Oct-04 FIO/DS 15

16 Wintertime? 19-Oct-04 FIO/DS 16

17 This is now Quattorized + Lemonized Rely on Operator and SysAdmin teams Operated in same way as PC farms Getting more out of suppliers BIOS upgrade necessary for PXE enabling BTW: most applies to tapeservers as well 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 17

18 What s next? New hardware 360 TB SATA in a box, 2 different suppliers 140 TB FC attached external SATA disk arrays New software SLC3, RHEL 3 New CASTOR stager New challenges Oracle SAN setup Alice data challenge 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 18

19 Conclusions A lot of work has been done to Stabilize Hardware and Software Automate + hand over basic operations Integrate into standard work flows Get more out of available hardware Achieved pro-active data management 19-Oct-04 Jan.van.Eldik@cern.ch FIO/DS 19

20 Useful links Standing on the shoulders of giants Tim Smith CHEP Helge Meinhard CHEP Peter Kelemen CERN IT After C5 Jan Iven HEPiX 2004 Edinburgh 19-Oct-04 FIO/DS 20