Elastic Management of Cluster based Services in the Cloud

Transcription

1 First Workshop on Automated Control for Datacenters and Clouds (ACDC09) June 19th, Barcelona, Spain Elastic Management of Cluster based Services in the Cloud Rafael Moreno Vozmediano, Ruben S. Montero, Ignacio M. Llorente Distributed Systems Architecture Research Group Universidad Complutense de Madrid 1/23

2 Contents 2/19 Introduction Elastic management and implementation of clustered services Experimental results Conclusions Demo

3 Introduction Goals Deployment of clustered services on top of a virtualized infrastructure consisting of: Local (in house) virtual infrastructure Cloud resource provider (Amazon EC2) VM manager (the OpenNebula engine) Service Users Virtual service infrastructure Service Layer Capacity Manager OpenNebula Core OpenNebula XEN / KVM Drivers EC2 Driver Cloud Provider (Amazon EC2) Infrastructure Layer Local Infrastructure 3/19

4 Introduction Motivation of the work The proposed architecture offers several benefits The virtualization layer allows to separate the infrastructure management (VM provisioning) from the service management Elastic provisioning of virtual resources on demand, to adapt the infrastructure to the service requirements. Fully transparent for the service itself, and independent of the type of service The integration with the cloud provides additional capacity to the virtual infrastructure 4/19

5 Introduction The VM Manager (OpenNebula) Capacity Manager OpenNebula OpenNebula Core XEN / KVM Drivers EC2 Driver OpenNebula Core: manages the life cycle of a VM by performing basic VM operations (e.g. deployment, monitoring, or termination) Capacity Manager (scheduler): Adjusts the placement of VMs based on a set of configurable provisioning policies Virtualizer Access Drivers: Pluggable drivers that provide an abstraction of the underlying virtualization layer (Xen, KVM, EC2), by exposing the basic functionality of the hypervisor or cloud interface 5/19

6 Contents 6/19 Introduction Elastic management and implementation of clustered services Experimental results Conclusions Demo

7 Elastic manag. and implement. of clustered services Benefits Elastic cluster capacity The capacity of the cluster can be modified by deploying (or shutting down) virtual cluster nodes on demand Cluster partitioning The available physical and cloud resources can be used to deploy VMs bound to different cluster based services This approach isolates the different workloads and the performance assigned to each cluster Heterogeneous configurations The VMs of a service could have multiple software configurations The different service images can be maintained with a minimal operational cost ( install once deploy many approach) 7/19

8 Implementation of clustered services in the cloud Implementation of a computing cluster Batch job submission system Sun Grid Engine (SGE) Cluster components Front end node: SGE master Back end nodes: SGE workers Job submission requests Front end server node (SGE master host) Service LAN Virtualized back end nodes (SGE worker nodes) OpenNebula Hypervisor (XEN) Hypervisor (XEN) Amazon EC2 Physical resource pool 8/19

9 Implementation of clustered services in the cloud Implementation of a web server cluster Based on NGINX software Cluster components Front end node: NGINX reverse proxy Back end nodes: NGNIX web servers HTTP Client HTTP Client Internet HTTP Client Front end server node (NGINX reverse proxy) Service LAN Virtualized back end nodes (NGINX web servers) OpenNebula Hypervisor (XEN) Hypervisor (XEN) Amazon EC2 Physical resource pool 9/19

10 Implementation of clustered services in the cloud The network infrastructure (i) The service LAN Every virtual node (both local and Amazon EC2 nodes) communicates with the front end through the service LAN (using private addresses) The local (in house) virtual nodes Are directly attached to the service LAN by means of a virtual bridge configured in every physical host The remote (Amazon EC2) virtual nodes Are connected to the service LAN by means of a virtual private network (VPN) tunnel The VPN tunnel is implemented with the OpenVPN software 10/19

11 Implementation of clustered services in the cloud The network infrastructure (ii) Amazon EC2 OpenVPN clients Back end Node Back end Node Back end Node OpenVPN Tunnels Internet Connection Bridge Front end Node + OpenVPN Server Physical Host 0 Bridge LAN (Gigabit Ethernet) Bridge Bridge Bridge Bridge Back end Node Back end Node Back end Node Back end Node Back end Node Back end Node Physical Host 1 Physical Host 2 Physical Host 3 Physical Host 4 11/19

12 Contents 12/19 Introduction Elastic management and implementation of clustered services Experimental results Conclusions Demo

13 Experimental Results 13/19 Computing cluster performance 0L+4R 4L+0R 4L+4R 4L+8R 5,0E 02 4,5E 02 4,0E 02 3,5E 02 Throughput (jobs/sec.) 3,0E 02 2,5E 02 2,0E 02 1,5E 02 1,0E 02 5,0E 03 0,0E Jobs

14 Experimental Results Web server cluster performance Throughput (requests/s) 300 HTTP requests (static file 1KB) L 8L 4L+4R 4L+8R 4L+16R Cluster configuration Throughput (requests/s) HTTP requests (static file 10KB) 4L 8L 4L+4R 4L+8R 4L+16R Cluster configuration Throughput (requests/s) HTTP requests (static file 100KB) Throughput (requests/s) HTTP requests (static file 1000KB) 0 4L 8L 4L+4R 4L+8R 4L+16R 0 4L 8L 4L+4R 4L+8R 4L+16R Cluster configuration Cluster configuration 14/19

15 Contents 15/19 Introduction Elastic management and implementation of clustered services Experimental results Conclusions Demo

16 Conclusions We have analyzed the deployment of two cluster based services on top of a virtualized infrastructure, with the capacity of integrating external cloud resources. Computing cluster Web server cluster This approach separates the resource provisioning from the service management, and provides important benefits: Elastic capacity to adapt the cluster to its dynamic workload Cluster partitioning to isolate it from other running services Heterogeneous configurations tailored for each application class Performance results show A sustained performance increment when adding a growing number of cloud nodes to the cluster That proves the feasibility of the proposed architecture and provisioning model, and its capacity to support service elasticity. 16/19

17 Contents 17/19 Introduction Elastic management and implementation of clustered services Experimental results Conclusions Demo

18 Elastic Management of Cluster based Services in the Cloud 18/19 THANK YOU FOR YOUR ATTENTION QUESTIONS?