Report from the HEPiX Workshop Fall 2013

Transcription

1 HEPiX Fall 2013 University Of Michigan, Ann Arbor from 28 October 2013 to 1 November 2013 Report from the HEPiX Workshop Fall 2013 Three individual views Jan Engels Thomas Finnern Peter van der Reest IT Seminar 26 November 2013

2 The Meeting, People and Miscellaneous > HEPiX Fall 2013 Workshop/Meeting/Conference > High Energy Physics Information exchange > We had 115 attendees (47 US, 48 EU, 3 Asia, 2 Australia) > including 27 from 8 american universities and 15 (!) from companies J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 2

3 Event Outline > The meeting contained all usual HEPiX tracks 65 contributions Scheduled total duration: 26 hours Rich and very interesting program High quality of presentations > 3 DESY Talks Wolfgang Friebel: DESY Site report (Track Site Reports) Jan Engels: Building a Puppet DESY (Track Basic IT Services) Patrick Fuhrmann: dcache news (Track Storage and File Systems) > Helge Meinhard: Wrap Up J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 3

4 Scientific Programme (Workshop Tracks) > Site reports 20+ Sites reported > Security & Networking covers issues related to networking technologies and associated monitoring tools, including the planning for and the use of IPv6, as well as network, system, Grid and cloud security. > Storage & Filesystems discusses everything related to storage technologies and hardware and to file systems, in particular distributed file systems. > Grid, Cloud & Virtualisation is about grid services, cloud technologies and experiences and virtualisation as an underlying technology for dynamic resource provisioning. > Computing & Batch Services is about computing technology evaluation, eg. benchmarking activities, GPU experiences, batch services... > IT Facilities & Business Continuity discusses news on and experience with computer rooms, cooling, electricity, facilities monitoring as well as business continuity. > Basic IT Services covers the services that are needed as a layer between the facilities and the user-facing services, for example configuration, IaaS, CMDB, monitoring. > End-user IT Services & Operating Systems discusses the services that are user-facing such as mail, version control, issue tracking, messaging, service management etc. unless covered in more specific tracks. It also covers operating systems. > Miscellaneous if you have something that doesn't fit in any other track... J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 4

5 Tracks and Trends > Security & Networking: 9 total, 3 on federations > Storage and Filesystems: 10 total, 3 on AFS, 2 on CEPH, WD vendor talk > Grid, Cloud & Virtualisation: 7 total, 3 on production private clouds > Computing & Batch Services: 6 total, 4 on batch systems HTCondor > IT Facilities & Business Continuity: 3 total > Basic IT Services: 8 total, 4 on Puppet, 2 on log analysis > End-user IT Services & Operating Systems: 3 total J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 5

6 Batch (1) > HEPiX Working Group > Site Stop the working group but keep updating the wiki, add HTCondor in Nikhef: Work on Grid Engine as alternative batch system (to LSF) Caltech CMS T2: Grid Engine to be replaced by Condor BNL: Condor on sl6 DESY: HH: 7700 Slots Torque/Maui + SoGE ZN: CTA, H.E.S.S 1600 Slots UGE RAL: HTCondor to Torque/Maui, CREAM to ARC CE The University of Wisconsin Madison CMS T2: HTCondor KIT: Univa Grid Engine 10K Job Slots UK GridPP T2: Condor, PBS/Torque, SGE FERMI: Condor and PBS IHEP (T2): Torque/Maui extra script for handling gpu support RAL T1: existing Torque/Maui batch system and preparing a new system J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 6

7 Batch (2) > CERN Evaluation: Future of Batch Processing at CERN Talk: Jerome BELLEMAN Increasingly challenging scalability and flexibility needs for Agile Infrastructure Demands: 30k-50k nodes, dynamic in/out, Hz dispatch, 100Hz query scale Features: Grid, Kerberos, Accounting, Host Normalisation, Fairshare, Commercial Applications, SLURM , HTCondor SoGE 8.1.3, LSF 8/9 Over LSF limits... SLURM too young SoGE fast, a bit rough HTCondor mature and promising > UNIVA Vendor Talk: Grid Engine: One Roadmap Owner of GE rights/properties with back version support 6.2u567 Support, continued product development: 28 releases & updates since 6.2u5 Current 8.1 since Aug '12: HA, Performance (core binding, resource classes, urgency) Version 8.2 in Q1, 2014: Windows native, Cgroups, Read only qmaster thread (qstat), drmaa v2, Performance No strong focus on Grid Engine Community outside UNIVA J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 7

8 Batch (3) > Batch > HPC 95 % Std batch facilities Trend: Partitionable slots (a batch resource able to be sliced along a variety of dimensions, from RAM to CPUs, to disks or even GPUs) New Champion: HTCondor 5 % special requirements Parallel execution, disc I/O, memory channel saturation, latency/speed, communication overhead Same equipment, 10Gb, single non blocking switch Future: MPI scaling to nodes > Clouds Dynamic Infrastructure mostly at bigger sites OpenStack (CERN) und OpenNebula (Fermilab) Loosing only a few percentage of performance (mainly data access) Winning management flexibility Kiosk Systems on Top (e.g. CERN) Peak Load from Batch (e.g. RAL to Stratus Lab) J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 8

9 Ann Arbor Impressions (1) J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 9

10 Configuration Management > Atlas Great Lake Tier 2 University Michigan (AGLT2) Transition to SL6 Cobbler used for minimal kickstart, most configuration done with CFEngine3 Evaluated Puppet/Foreman but prefer staying with CFEngine due to expertise on site > BNL RACF Migrating to Puppet 3 Using Cobbler + Puppet (2.7) for managing all RedHat/SL virtual hosts Testing Foreman as Cobbler replacement Testing Katello and MCollective > University of Wisconsin Madison CMS Tier 2 Migrated from CFEngine to Puppet this summer. J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 10

11 Configuration Management > GridPP Transition to SL6 and Puppet Puppet group established to share expertise with the rest of the community > IHEP Migrating Quattor to Puppet Start: 1 Oct, End: 31st Dec!! Using Puppet Foreman 1.3 J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 11

12 Configuration Management > CERN Migrating Quattor to Puppet 3 (until end of 2015) 4000 hosts in production Aiming to manage VM's in the future 150!! persons managing configurations Introduced new workflow Separate git repository for each Puppet module MCollective used in production JIRA/Git increasingly popular, CVS finally about to be stopped J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 12

13 Configuration Management > Fermilab ~2100 hosts running Puppet 134 puppet modules + 27 custom facts Running puppet (apply) locally on client machines Installing yum packages with exec (for using yum enablerepo ) > Fermilab ~ 1200 hosts running SLF5 Migrating all hosts to SLF6 Cobbler for system installation - Puppet for configuration management Running Puppet3 + Hiera + PuppetDB Puppet Environments + Git Proven to be extremely useful! J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 13

14 Configuration Management > GridKa Middleware services used as testbed for puppet > Started in 2012 Starting to draw up plans for puppet migration > DESY > Git, Foreman, MCollective, Transition to SL6 Migrating to Puppet 3 Dedicated Puppet Talk: > Building a Puppet DESY (see next slide) J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 14

15 Current Puppet DESY J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 15

16 Configuration Management Summary > Most sites migrating to SL6 > Puppet Migration to Puppet reported on 7 (out of 16) sites 1 Site preferred CFEngine3 over Puppet 3 talks dedicated to Puppet Most sites using Puppet 3 > Cobbler widely used for provisioning > MCollective and Foreman mostly under evaluation J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 16

17 Scientific Linux 5 > Red Hat 5.10 released on Sept > SL 5.10 Beta-1 released on Oct > Significant changes: MySQL 5.0 no longer supported! > Packages provided to migrate to MySQL 5.5 > Can't migrate back afterwards! MySQL 5.5 is a Software Collection style package J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 17

18 Scientific Linux 6 > RHEL 6.5 released on Nov > SL 6.5 currently being built... > No UEFI Secure Boot support on RHEL6 and therefore SL6 > Features Better support for PTP (Precision Time Protocol) Better support for SSD's Newer openssl Native hooks for GlusterFS J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 18

19 Scientific Linux 7 > According to Red Hat, the public RHEL Beta should still be released in maybe early 2014? > Expected to look somewhat like Fedora 19 > RHEL7 will use systemd NetworkManager Gnome3 (classic mode!!) XFS will probably be prefered over EXT{2,3,4} J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 19

20 Impressions Ann Arbor / Detroit (2) J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 20

21 Storage & Filesystems Storage Track convened by Arne Wiebalck and Peter van der Reest one day, 10 talks one active working group: bit preservation chaired by German Cancio and Dmitry Ozerov Collecting and sharing knowledge on bit preservation across HEP (and beyond) Provide technical advise to Recommendations for sustainable (distributed) archival storage in HEP Many labs still looking for large, cheap storage some need POSIX some want Cloud FNAL has performed evaluation: dcache, EOS, HDS to cover all areas no more Lustre, no HADOOP, no Isilon J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 21

22 OpenAFS and IPv6 > discussion was started at last HEPiX on the importance of having OpenAFS ready for IPv6 > contrary to feeling in meeting then, sites did not respond or said that the problem was not perceived as acute > in parallel talks with companies w/ OpenAFS support offerings we held both estimated 6 months development and some 3-6 months of testing and debugging both have different approaches cost could be shouldered by HEP community alone, but could leave HEP stranded on a private source branch alternatively start with a new, backward compatible product called YFS but would introduce annual costs > current position: wait and see while looking at dual stack scenarios that could help out J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 22

23 OpenAFS status > mid year: focus on security > since: continuing effort on optimizing and tuning caching behaviour threading > support for Windows8 on Surface (ARM) > current stable release > 1.6.6pre has been released for testing > OpenAFS Foundation has been created way of channeling funds for development long time legal issues with IBM solved J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 23

24 YFS > product development started after pressure from HEP(iX) > Dual protocol stack clients and servers AFS stack for compatibility with IBM AFS and OpenAFS clients and servers YFS stack for enhanced functionality > Focused improvements: Security (K5, AES256, upgradable, identities, ACLs CB channel protection) Networking (new Rx implementation, IPv4 and v6 capable, WAN performance improved) Scalability (16k RPCs in parallel, endless volumes, directories and files) Ease of Use (highly integrated in Win and Mac) Code Quality (refactoring, cmdline tools are wrappers around libs, test suites for all libs) > mixed deployment possible > available now for RHEL, Fedora, Debian, MS Server2003+, OSX 10.8+, ios J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 24

25 Mass Storage > Report from FNAL on their solution for small files in ENSTORE Library Manager Director Encp Encp client client Disk Library Manager Disk Disk Mover Mover Cache File Clerk Policy Engine Server & Migration Dispatcher Tape Storage (Enstore) Migrator Migrator > condensed form: create container files and have a policy engine do work in the background > in production since 2012 J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 25

26 dcache update > dcache project well established and secured funding for coming years > used world wide in all sorts of organisations, mainly HEP > workshops to evangelize well received > work on NFS4.1/pnfs continuing > work on WebDAV gateways > Small File Support for Tertiary Backends Support of HDF5, Nexus file formats XAML, Oauth Support Cloud Data Management Interface > Object Stores > new: the scientific storage cloud separate presentation by Patrick? J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 26

27 CEPH > both CERN and US ATLAS Midwest Tier 2 presented on setups and experiences > CERN has installed evaluation cluster after feasibilty study proved positive > 3 PB on 48 server w/ 24 3 TB disks, 5 monitoring servers > fully puppetized, integrated in CERN monitoring > early adopters helping by using in production OpenStack AFS/NFS DPM OwnCloud, Zenodo > USATLAS still evaluating > The killer app for Ceph would be to build upon it a general purpose network file system J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 27

28 Ann Arbor Impressions (3) J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 28

29 Logging, Monitoring, Accounting, > BNL: Using control system tools for operation and debugging Gather information from multiple places Allow the end user to create ad-hoc ways to mix and match data without the need of writing programs or web pages Different display modes Node i/o load squares in bubbles Room scapes with drill down and freely configurable elements > GSI: Logstash and ElasticSearch deployment scenario Use cases for Logstash and ElasticSearch Logs of different services: Grid Engine, SSH, Apache and Cisco Firewalls, among the others Parse, index and analyze logs of different services Grid Engine, SSH, Apache and Cisco Firewalls, among the others Kibana (V3) as an HTML plus Javascript interface to improve the analytics in ElasticSearch. Logstash: log, parse and store Logstore contains normalized and originaldata Dashboard follows dynamic reselection of e.g. time Logstash Redisp Kibana 3 html/javascript (kibana 2 Ruby) No access rules so far > CERN: Security Be aware of J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 29

30 Security (1) Interactive Session with Romain WARTEL > No public slides > This is an interactive technical session aimed at system administrators, and presenting several Linux tools and tips that can be used to both drastically reduce the chance of root compromise, as well as increase the amount information available during forensics. > Be more secure in 6 steps > Motivation: Large distributeted community in HEP > The are always compromised accounts > Trace patch to get rid of basics decent atttacks com with 0-day exploits valid user with 0-dya or stolen admin credentials or... may day months/years keep records centrally uncomprimised user actions + detailed network records (auditd, snoopy: user mapping) > standalone incidents are history > Security Alerts on user if come in from different net location > Ebury: ssh malware, malware, domain names, passwd collectors, routine maintenance,... J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 30

31 Security (2) Update from Romain WARTEL > No public slides > It describes the main vectors of compromises in the academic community and presents interesting recent attacks. It also covers security risks management in general, as well as the security aspects of the current hot topics in computing, for example identity federation and virtualisation. > Paradigm change: Local users -> User communities/federations Firewall -> Tracebility Mal Users -> Mal Orgs Selling malware / $ $ $ Gov-Class Attackers scale and scope, direct vendor access no surprise nsa not the only one impact on HEP labs: who owns routers, data privacy, have experts,... cooperation is crucial e.g. CERT > Don't trust any J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 31

32 CERN Reports Highlights > Open Days 70k people (20k people in tunnel) > Wigner Data Centre now fully operational (400 Cloud nodes with LSF) Agile Infrastructure > Cloud Status (openstack.cern.ch) OpenStack Grizzly: 700 Hypervisors + 100/week => (2015) Ceph + Nova cells: distributed, scaling, hierarchical KVM and Hyper-V share the same hardware (Hyper-V limited in console acc and resizimg,...) EC2(CMS,Atlas, NA61) und Nova-api > Mobility at CERN wireless-capable devices = traditional cabled network WIFI: 7000 unique devs / day BYOD strategy and the integration of eduroam. Large conference rooms and underground facilities will also be addressed. BYOD: Individual devs can reduce overall service, Avoid 2.4GHz Conference rooms: Capacity is key concern Underground caverns: Coverage is key concern J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 32

33 and other exciting things > Identity Management in Future Scientific Collaborations id + auth, trust relationship, user/user isolation, persistance user data complex roles,..., technology, HPC, cloud(hsc), + GRID cert free job submission (e.g. OSG Authentication Model) native data access (e.g. Datacloud over dcache) > Experience CSC: Everybody outsources, in rural location all to the same man CSC: Remote management works for computers not for human BNL: TCO Datacenter $0.04/h(Dedicated) vs. $0.12/h(EC2) J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 33

34 Board Meeting > Current and next meetings > Working groups IPv6: see report; plan to hold a 2-3 day workshop Storage: terminated no final report as Andrei couldn t attend Benchmarking: New SPECcpu next October, plan to start working with experiments early Batch: stop the working group but keep updating the wiki, add HTCondor info Configuration management: co-chairs not able to attend; needs more visibility Bit preservation: focus on technical advice on bit perservation Energy efficiency: keep open for Annecy > Web site needs attention help by DESY much appreciated > North-American co-chair Sandy s 2 nd term ends fall 2014 Search to begin at Annecy, election in board meeting during fall meeting J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 34

35 Next meetings > Spring 2014: LAPP Annecy, France May confirmed Same tracks as this time > Fall 2014: University of Nebraska interested Need final site approval Dates to be determined > Spring 2015: Candidate site in Europe identified Proposals always welcome J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 35

36 Thank you for listening J. Engels, T. Finnern, P.v. d. Reest Report from the HEPiX Workshop Fall Page 36