HP Helion CloudSystem 9.0: Using VMware ESXi Clusters as Compute Nodes

Transcription

1 HP Helion CloudSystem Enterprise and Foundation Software HP Helion CloudSystem 9.0: Using VMware ESXi Clusters as Compute Nodes This white paper provides an overview of VMware ESXi support in CloudSystem 9.0, including best practices and recommendations for using ESXi clusters with CloudSystem. Overview HP Helion CloudSystem is based on the Helion OpenStack distribution of OpenStack cloud software. CloudSystem integrates hardware and software to deliver core Infrastructure as a Service (IaaS) provisioning and lifecycle management for compute, network, and storage resources. CloudSystem supports ESXi clusters, Hyper-V, and KVM hosts as compute nodes. The following diagram shows the key components that enable CloudSystem to use ESXi clusters.

2 Nova, the OpenStack compute service, is used to deploy virtual machine instances into ESXi clusters. Instances are provisioned on ESXi clusters if the image being used in the provisioning request is a VMDK image. Each ESXi cluster appears as a single large compute node to the Nova scheduler. The scheduler selects a cluster that has sufficient capacity, and then Nova provisions the instance on that cluster. DRS on the cluster helps place the VM on the most suitable host within the cluster. CloudSystem enables ESXi support in Neutron, the OpenStack network service that supports only ESXi distributed vswitches and security groups.. Glance, the OpenStack image service, stores VMDK images in the Glance repository. Nova copies them to the cluster s shared datastore when a new instance is deployed. Cinder, the OpenStack block storage service, supports block storage on VMware datastores using the Openstack Cinder VMDK driver. Cinder volumes created using the VMDK driver are attached to instances on ESXi via datastores. During volume attach, the volumes are created on the clustered datastores on the host where the VM instance is running. This allows for volumes to be available when the VM is migrated across hosts in the cluster. Volumes can be used as data disks or boot disks. The option to boot from an OpenStack Cinder volume is supported. The following is a summary of the various networks shown in the diagram. 2

3 Network Number Purpose Connected to Data Center Management Network 1 This network connects virtual appliances to HP 3PAR, VMware vcenter Server, VMware vcloud Networking and enclosures. Cloud controllers, Management appliances, OVSvApp VMs, Enterprise appliances, VMware vcenter Server Cloud Management Network Consumer Access Network 1 This network connects to Management appliances, Cloud controllers, Monitoring appliances, and update appliance, KVM, Hyper-V compute nodes. This is a private network. 1 This network provides cloud users with access to the CloudSystem Portal in Cloud controller and the Marketplace Portal in Enterprise. Cloud users on this network are not able to access other CloudSystem appliances managed by the cloud administrator. External Network 1 This network allows cloud end users to attach public IP addresses to their provisioned virtual machine instances. KVM compute nodes, Hyper-V compute nodes, Cloud controllers, Management appliances, OVSvAPP VMs, Update appliance, Monitoring appliances, SDN network node (VXLAN case) Cloud controllers, Enterprise appliances Cloud controllers Provider Networks and/or Private Networks At least 1 A Provider Network is a data center network routed through the existing data center infrastructure. A Private Network is created from a pool of VLANs. Both networks support instance communication. Cloud Data Trunk OVSvAPP VMs Compute nodes CloudSystem uses the Cloud controllers to run the Nova compute agent with a vcenter driver. The OVSvApp VM runs the neutron Layer 2 agent. These agents perform all lifecycle operations on VM instances. After registering a vcenter Server with CloudSystem, all the clusters are automatically imported. You can activate one or more clusters managed by that vcenter Server. The activation process will deploy an OVSvApp VM on each host in the cluster, create necessary switches, and deploy a compute agent on Cloud controllers. CloudSystem 9.0 supports three vcenters, with each Cloud controller managing one vcenter. 3

4 Planning for setup of ESXi clusters Before setting up new clusters or reconfiguring existing ESXi clusters with CloudSystem, take the following actions. Pick a suitable cluster size (best practice is 10 to 16 hosts per cluster). As additional compute capacity is required, set up additional clusters rather than expand existing clusters. Existing VM instances that use cinder volumes will not be able to move to newly added hosts when clusters are expanded. A cluster can have 256 volumes attached to its hosts. Consider the 256 LUN limit per cluster when sizing. See the Storage requirements section for details about the 256 LUN limit. Choose a suitable shared datastore size. The size is based on the typical boot volume size that is determined by the flavor and the expected number of VMs per cluster. Boot volumes on ESXi clusters are thick-provisioned. Instead of using shared datastores, best practice is a single large shared datastore for better deployment times and optimal storage usage. Identify the free NIC for all the hosts in the cluster and pass it on as cloud trunk interface while activating the cluster, or identify the Cloud Data Trunk. Set up the Cloud Data Trunk on all ESXi hosts in all clusters used by CloudSystem and pass it as a cloud trunk during activation. Note that the same NIC must be connected to the cloud data trunk. CloudSystem supports only Distributed vswitches (DVS). Basic networking is provided with security groups. When using a pre-existing cluster, best practice is to remove all VMs from the cluster before activating it in CloudSystem. Creating compute nodes To view a detailed step-by-step process to set up the Compute nodes, search for Creating ESXi compute clusters in the HP Helion CloudSystem Administrator Guide. Registering vcenters You can register up to three vcenters from the Integrated Tools screen in the Operations Console. When registering vcenters, avoid using special characters in the name. During registration, the VMFS block driver is configured automatically to enable creating VMDK volumes. Activating ESXi clusters Activating an ESXi cluster allows Nova to use the cluster as a compute node for provisioning VM instances. The activation process performs a set of validations that ensure the cluster meets the prerequisites to become a compute node. Each cluster has at least one host in a connected state. Empty clusters cannot be activated. Clusters across datacenters can be activated. The cluster names must be unique. The cluster has at least one shared datastore accessible to all hosts in the cluster. Clusters have DRS enabled. Cluster names cannot contain special characters. 4

5 If any of the prerequisites are not met, the activation process fails and the conditions that were not met are displayed. Complete the required conditions, and then retry activation. De-activating ESXi clusters A cluster can be deactivated from CloudSystem only when there are no instances provisioned by CloudSystem in the cluster. If the cluster needs to be decommissioned, first deactivate the cluster from CloudSystem, then remove all hosts in the cluster and delete the cluster. Summary and limitations of OpenStack operations The following OpenStack operations differ from what is generally described in OpenStack documentation. Snapshots The boot disks in ESXi are full thick disks with a size defined by the flavor. When a snapshot is created from a VM instance, the boot disk could be much larger than the Glance image that was originally used to provision the VM. This can cause image upload and subsequent download of a snapshot to increase proportionally. Volume attachments When volumes are attached to instances running on ESXi, the volumes are not automatically mounted in the guest OS. The mount path follows a Linux OS convention irrespective of the VM guest OS. The specified mount path determines the resulting disk unit number/index. Passing device name is not mandated. Detect the disk using the Guest OS specific command to rescan disks. For example, on Fedora: echo "- - -" > /sys/class/scsi_host/host2/scan fdisk -l This command shows the newly attached disk where a file system can be created and mounted. Resize OpenStack Nova resize (ESXi or KVM) is not supported in CloudSystem. Migrate/Live Migrate Live migrate via the OpenStack Nova CLI/API is not supported on ESXi clusters. However, VMs can be migrated within members of the cluster from vcenter Server, either manually or automatically via DRS. Pause/Un-pause OpenStack Nova pause and un-pause are not supported on ESXi. These operations have no effect on instances. Interface attach/detach Additional network interfaces cannot be attached to an instance after it is created. 5

6 Compute requirements and notes The following list summarizes ESXi cluster configuration requirements for CloudSystem 9.0. DRS must be enabled for all clusters. Clusters must be consistently configured. o Shared datastores are visible to all hosts of the cluster. o Consistent network and storage connectivity is configured. o VMware vmotion is configured. Disconnect any datastores that are available to only select hosts of the cluster. Enable HA on the cluster, if needed. Provision all VMs in the root resource pool of a cluster. CloudSystem ignores any other ESXi resource pools configured in a cluster. Cluster operations New hosts may be added or removed from the clusters used by CloudSystem. For detailed instructions, see the HP Helion CloudSystem Administrator Guide. To add a new host, make sure that the new host meets all of the requirements of the cluster (DvSwitches are present and there is a common shared datastore). Put the host in the maintenance mode and run the EON CLI mentioned in the Administrator Guide. To delete a host from a cluster, put the host in maintenance mode and evacuate all VMs from the host. Next, delete the host. To tear down the cluster, deactivate the cluster in CloudSystem, and then delete the cluster. If the static IPs are used for OVSvApp VMs, they are not cleaned up automatically. To add additional compute capacity, best practice is to add a new cluster instead of adding hosts to an existing cluster. Use the hosts available to create clusters instead of using many small clusters that you can grow gradually to increase compute capacity. Networking requirements and notes Cloud system supports both VLAN and VxLAN network underlay types with Neutron virtual router configuration, CVR (Central Virtual Router). ESXi VLAN Network Topology with OVSvApp The following diagram shows the network topology with ESXi hosts when deployed with VLAN overlay. All management and compute hosts are configured with VDS. On each ESXi host, an OVSvAPP VM is activated to support Neutron security group features. Three trunk networks are included in CloudSystem for ESXi, KVM, or Hyper-V environments: DataTrunk (Provider/tenant): A VLAN trunk for VLAN overlay and a dedicated IP network for VxLAN. All provider and tenant networks are provisioned in this trunk. 6

7 Management Trunk: All cloud management networks are provisioned in this trunk. Storage Trunk: A dedicated trunk for all storage traffic, including block and object storage. At present, object store and block store networks are provisioned in this trunk. This trunk is optional if the deployment does not require Swift and VSA networks. For VxLAN deployment, instead of VLANs trunk wired to provider/tenant networks, only a single VLAN network is needed. Network provisioning in ESXi When a VM is provisioned on an ESXi cluster, it connects to a specific set of networks, as defined in the provisioning request. The Neutron services running on Cloud controllers determines the VLAN that is used for each network and tells the SDN agents running on OVSvApp VMs to create a port group with the appropriate VLAN. The following is a high-level summary of network connection configuration for provisioned VMs. A VM is created in a cluster without any NICs. The Neutron agent detects the VM and determines the required networks. The Neutron agent creates port groups (if not already created) for the networks. The name of the port group(s) is the OpenStack Neutron network UUID. All ports on this port group are disabled when the port group is created. NICs are added to the VM and connected to the port groups set up by the SDN agent. The Neutron agent updates the security group rules associated with the port group. 7

8 The port corresponding to the VM is enabled. Port groups are deleted when the last VM connected to the port group is deleted. Any VMs created out of band and connected to the Cloud Data Trunk vswitch are not connected because their ports are disabled. Use other switches for VMs created outside of CloudSystem. Security group implications All VM instances on Provider Networks, that belong to one or more tenants, can reach any other VM on that Provider Network (tenant boundaries do not apply). Access control is not enforced. A VM instance on a Private Network can reach any other VM on the Private Network, even if it was created by another user of the same Private Network. Since Private Networks dedicate a specific VLAN for a tenant, tenant isolation is provided. Storage requirements and notes Boot image storage and caching Shared datastores in ESXi clusters store the Glance images used to create (boot) new VM instances. An image cache is created on each shared datastore when a Glance image is used for the first time. A copy of the Glance image is made for each flavor used (according to the boot disk size specified in the flavor). The boot disks of provisioned VM instances are created as linked clones and the images in cache are used as the base disks. All boot disks are thick provisioned. The VM instances are linked clones, and they may not use as much space as that allocated by the flavor. The OpenStack Nova scheduler computes available disk space based on disk allocation and not actual utilization. You may see more free space available than what CloudSystem shows as allocated. Oversubscription ratios can be tweaked to alter actual utilization. However, instead of changing oversubscription to increase utilization, create the volumes backing the datastores using thin provisioning and manage disk oversubscription at the HP 3PAR that is backing the volumes. You can add new datastores to the cluster, but make sure that the new datastore is available to all hosts in the cluster. Creating Images for ESXi For information on creating VM boot images for ESXi, see the Creating Microsoft Windows Virtual Machine Images for Use with ESX Compute Clusters in HP CloudSystem white paper in the Enterprise Information Library. VM boot process and image transfers When a VM image is used for the first time on an ESXi cluster, the Nova agent downloads the image from the Glance repository and copies the image to a cache directory on the shared datastore with the most space in the target cluster. When the download completes, the VMDK image is moved from the temp directory to a base directory. The image is copied to the shared datastore by the vcenter Server HTTP transfer capability. The following diagram shows the image transfer path from the Glance repository to the image cache on the shared datastore. 8

9 Cloud Mgmt Network CC3 CC1 CC2 Neutron Neutron Neutron Nova Compute Glance Cinder volume Management Cluster Datacenter Management Network vcenter Cloud Data Trunk External Network Managed Cluster iscsi or FC network 3Par Storage System ESX Host ESX Host ESX Host OVSvAp p OVSvAp p OVSvAp p Volumes for Shared Volumes Datastores for Shared Volumes for Shared Datastores Volumes Datastores for Shared Datastores Glance Image Cache Cluster Shared Datastores LEGEND Datacenter Management Network Cloud Management Network Cloud Data Trunk External Network iscsi or FC network For all subsequent requests to create (boot) VM instances from an image that is in the image cache, the image is no longer downloaded. The image that is present in the cache is used as a base disk and a linked clone is created. Thus, any subsequent VM instances are created using an image in the cache. This is much faster because the image download is avoided. Snapshots VM snapshots result in a new Glance image that is created from a VM instance. The image is created from the VM s boot disk and the size is determined by the flavor. It is important to keep in mind that VM boot disks are thick provisioned. This size could be much bigger than the original boot image that was used to provision the VM. The time required to upload the snapshot into Glance is proportionally higher. Cinder Block Storage Use the VMware VMDK block storage driver to enable management of the OpenStack Block Storage volumes on vcenter Server managed datastores. Volumes are backed by VMDK files on datastores that use VMware-compatible storage technologies. Important: The VMware datastore property type must be NFS or VMFS for Nova to work correctly with CloudSystem. To verify datastore properties, log in to the vcenter Server Managed Object Browser using the vcenter Server IP address and credentials. Select Content, then Datacenters, then select the datacenter containing the activated cluster. Select the datastore, and then click Summary. The VMware VMDK driver connects to the vcenter Server, through which it can dynamically access all the data stores visible from the ESXi hosts in the managed cluster. When you create a volume, the VMDK driver creates a VMDK file on demand. The VMDK file creation completes only when the volume is subsequently attached to an instance, because the set of datastores visible to the instance determines where to place the volume. 9

10 The running vsphere VM is automatically reconfigured to attach the VMDK file as an extra disk. Once attached, you can log in to the running vsphere VM to rescan and discover this extra disk. Use the CloudSystem Console to configure block storage drivers and volume types. Register the vcenter Server hosting the driver on the Integrated Tools screen. You can configure one VMFS driver for each vcenter Server you register. Supported operations Data Disks Attach or detach a Cinder volume. Boot Disk Create a volume from a bootable Glance VMDK image. Learn more about HP Helion CloudSystem: Copyright Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein , November