The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud)

Transcription

1 15-19 April 2013, Bologna Italy HEPiX Spring 2013 Workshop Wojciech Ozga Faculty of Computer Science, Electronics and Telecommunication AGH University of Science and Technology in Krakow, Poland CERN, Geneve, Switzerland The CMS openstack, opportunistic, overlay, online-cluster Cloud (CMSooooCloud)

2 CMS HLT farm Requirements Network virtualization Hardware virtualization Conclusions OUTLINE

3 CMS CERN LHCb ATLAS ALICE CMS LHC Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #3

4 DAQ system is designed for taking up to 20TBytes of data per day CMS HLT farm Complex varied hardware different network schemas hundreds of OS configurations Large size 1264 nodes 50 switches Powerful cores 26.6 Tbytes of RAM 232 Tbytes of storage Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #4

5 CMS HLT farm vs Tier [0-2] CPU in HEP-SPEC06 HLT farm Tier0 Tier1 Tier2 HEP-SPEC06 195k 121k 150k 399k Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #5

6 Requirements Minimal Changes no special hardware reconfiguration No network reconfiguration Opportunistic Usage Quick switching from data taking to other tasks Flexibility in running different tasks VIRTUALIZATION No impact on data taking Nor during the setup phase Neither during data taking on Heavy Ion runs Full control Full control on hardware and OS no extra temporary sudoers Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #6

7 CERN CMS sites CDR The CMS Online Cluster Data networks Storage manager Readout, HLT Control Routed Service Network gateway gateway Private CMS networks - Service networks (per rack) ~3000 1Gbit ports - Data networks (VLAN s) ~4000 1Gbit ports - Central Data Recording (CDR) Network to Tier0 - Oracle RAC networks - Subdetectors networks CERN network firewall CERN network Internet Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #7

8 Network virtualization Integration of cloud network with CMS network GRE TUNNELING network virtualization CMS Network packet x.y/25 Cloud packet 10.1.a.b/16 + No router/switch re-configuration. No dependencies between production and cloud network. - Encapsulation overhead. NATing to CERN network required. Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #8

9 Network virtualization router Network A Network B GRE tunnel GRE encapsulation interface interface GRE encapsulation OpenVSwitch OpenVSwitch Cloud network Cloud network Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #9

10 CERN CMS sites CDR Data networks Storage manager Readout, HLT Control switch switch switch Routed Service Network gateway gateway /15 switch Router + NAT CERN network firewall Internet Physical network schema CERN Network Logical cloud network schema

11 Cloud approach Standard approach Hardware virtualization Configure physical machine to run specific task Number of configurations = Number of tasks x number of different hardware configurations Launch task specific Virtual Machine over hypervisor Number of configurations = Number of tasks + number of different hardware configurations Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #11

12 Compute-node Compute-node Compute-node Compute-node Compute-node MySQL Glance image service Keystone authorization service Network service RabbitMQ Scheduler service API (EC2, Openstack) Openstack Cloud Manager

13 Cloud Architecture OpenVSwitch NAT OpenVSwitch MySQL RabbitMQ Proxy Nova-Compute service Network service Scheduler service Keystone service API (EC2, Nova) Glance service K hypervisor 1 x Fat Controller Node 48Gbytes, 8 CPU, 8x1Gbit Ethernet 1 x Image Store High bandwidth Cache mechanisms 1300 x Compute node K virtualization OVS network virtualization Wojciech Ozga The CMSooooCloud "The best advice I know is 'Don't distribute.'" Adam Bien

14 CMS Cloud usage 8 th 12 th of October 2012 First big scale test: Folding@Home Cloud controls 1300 hypervisors 768 completed Work Units in 12h 21 st of December th of January 2013 Test of stability, running Folding@Home Stable run of 1000 s during 3 weeks Deployment of 250s in ~5min completed Work Units Since January 2013 Finalizing the integration as a CERN s GRID resource Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #14

15 Outlook Migration to the new version of OpenStack Grizzly release emphasize on high availability features Increase the network connectivity to CERN Tier0 bandwidth connection up to 40GBit/s Cloud network reconfiguration while we are in the long shutdown period benefit from VLANs at the expense of Openvswitch Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #15

16 Conclusions An Overlay Cloud layer deployed on CMS HLT cluster with Zero impact on data taking. Man power dedicated to cloudify the CMS HLT cluster was about ~1.5 FTE for ~6months CMS online share knowledge about cloud installation CMS HLT cluster computing power is being used in an opportunistic way leaving full control over cloud availability to DAQ CMS Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #16

17 Thank you. Questions? Wojciech Ozga The CMSooooCloud HEPiX Spring 2013 Workshop, April 2013, CNAF Bologna. #17