The VMware Software-Defined Data Center Foundation

Transcription

1 The VMware Software-Defined REFERENCE ARCHITECTURE VERSION 1.0

2 Table of Contents Overview Audience VMware Software Components Architectural Overview... 5 Virtual Management Networks... 5 Management Cluster... 7 Edge Cluster Compute Clusters Physical Component Details Compute Storage Network Software-Defined Data Center Component Details vsphere Data Center Design vsphere Data Protection NSX for vsphere Monitoring vrealize Operations Conclusion The VMware Validated Design Team About the Author...22 TECHNICAL WHITE PAPER / 2

3 Overview This reference architecture describes the implementation of a software-defined data center (SDDC) that leverages the latest VMware components and capabilities. The reference architecture is built on the Foundation VMware Validated Design. The Foundation VMware Validated Design is the core of all higher-level VMware Validated Designs. Foundation describes a scalable, resilient, best-practice configuration on which all additional functionality is layered. The Foundation configuration uses industry-standard servers, IP-based and VMware Virtual SAN storage, and software-defined networking to support a scalable and redundant architecture. The VMware Validated Design process gathers data from customer support, VMware IT, and VMware and partner professional services to create a standardized configuration that meets the majority of customer requirements. Internally, VMware engineering teams test new product capabilities, installations, upgrades, and more, against the standardized configuration. VMware and partner professional services teams build delivery kits based on this design, knowing that they are deploying with the best possible configuration. Customers planning do it yourself deployments also benefit from following this architecture, confident that future product upgrades, patches, and so on, have already been tested against a configuration identical to theirs. Audience This document will assist those who are responsible for infrastructure services, including enterprise architects, solution architects, sales engineers, field consultants, advanced services specialists, and customers. This guide provides an example of a successful deployment of an SDDC. TECHNICAL WHITE PAPER / 3

4 VMware Software Components This architecture uses the following VMware software components. PRODUCT VERSION DESCRIPTION VMware vsphere with Operations Management 6.1 VMware vsphere with Operations Management delivers VMware vsphere optimized for efficient server virtualization management by adding critical capacity management and performance monitoring capabilities used to deliver the SDDC. In this architecture, users leverage VMware vsphere with Operations Management Enterprise Plus Edition 6.1, which includes VMware vsphere Enterprise Plus Edition 6.0 U1, VMware vsphere Data Protection 6.1, and VMware vrealize Operations Manager 6.1. VMware vcenter Server 6.0 U1 VMware vcenter Server is the central platform for managing and configuring the VMware ESXi hypervisor. VMware vsphere Web Client is the centralized point of administration for compute clusters and all networking services provided by VMware NSX for vsphere. VMware Virtual SAN 6.1 Virtual SAN is radically simple, hypervisor-converged storage for virtual machines. It delivers enterprise-class, high-performance storage for virtualized applications, including businesscritical applications. VMware NSX for vsphere 6.2 NSX for vsphere exposes a complete suite of simplified logical networking elements and services including virtual switches, routers, firewalls, load balancers, virtual private network (VPN), QoS, monitoring, and security. Table 1. Components TECHNICAL WHITE PAPER / 4

5 Architectural Overview This design uses three cluster types, each with its own distinct function. It provides a management plane that is separate from the user workload (compute) virtual machines. It also leverages an edge cluster, which provides dedicated resources for network services such as VMware NSX Edge devices, which provide access to the corporate network and the Internet. Virtual Management Networks Table 2 provides a quick introductory overview of VMware NSX and networking terms and acronyms. TERM DEFINITION VLAN NSX for vsphere Virtual switch VPN MPLS A VLAN is used to partition a physical network into multiple distinct broadcast domains. NSX for vsphere is virtual networking and security software that enables the software-defined network. The NSX virtual switch abstracts the physical network and provides access-level switching in the hypervisor. It is central to network virtualization because it enables logical networks that are independent of physical constructs such as VLANs. A VPN can be used to connect networks to each other or a single machine to a network. Multiprotocol Label Switching (MPLS) is a scalable, protocol-independent transport mechanism. In an MPLS network, data packets are assigned labels. Packet-forwarding decisions are made solely on the contents of this label, without the need for examination of the packet itself. This enables users to create end-to-end circuits across any type of transport medium, using any protocol. Table 2. VMware NSX Overview This design utilizes NSX for vsphere virtual switches in the management cluster. VMware vrealize Operations has its own VMware NSX virtual switch and NSX Edge device. As new management or monitoring solutions are added, they too are placed on their own VMware NSX virtual switch with a corresponding NSX Edge device. The grouping of these virtual machines onto a virtual switch is referred to as a virtual management network. Virtual management networks are connected to both the vsphere management VLAN and the external VLAN via an NSX Edge device. Routing is configured between the virtual management network and the vsphere management VLAN. This enables the solution and virtual machines on the vsphere management VLAN, such as the vcenter Server instance, to communicate without exposing the management servers directly to end users. External connectivity to the business function of the virtual management network is provided by a load balancer s virtual IP on the external VLAN. External connectivity to the management function of the virtual management network is provided, as needed, by the method routing to the virtual management network using Ethernet, MPLS, VPN, jump hosts, or other means that best fits the customer s environment. Another option is to deploy the VMware NSX distributed logical router and to enable a dynamic routing protocol, such as OSPF, between the router and the top-of-rack switches. This enables access to all virtual machines on the virtual switches by advertising their IP addresses to the rest of the network. Virtual management networks still require NSX Edge devices to provide load balancer functionality. TECHNICAL WHITE PAPER / 5

6 Virtual management networking isolates management solutions from each other and from compute workloads. More important, it enables disaster recovery of the automation and monitoring stacks without having to change the IP addresses of the virtual machines. The virtual machines can be moved to precreated virtual switches in another site that has been configured the same as the primary site, enabling quick recovery of the solutions. Top-of-Rack Switches VLAN 1701 SVI VLAN 1680 SVI Static Routes / / / External VLAN (1701) /24 vsphere Management VLAN (1680) /24 vrealize Operations Virtual Switch /24 VM VM VM NSX Edge (Internal) (vsphere) (External) Figure 1. Example Using Static Routing TECHNICAL WHITE PAPER / 6

7 Top-of-Rack Switches External VLAN (1701) /24 vsphere Management VLAN (1680) /24 OSPF Distributed Logical Router vrealize Operations Virtual Switch /24 VM VM VM NSX Edge (Internal) Figure 2. Example Using Dynamic Routing Management Cluster The management cluster contains the management and monitoring solutions for the entire design. A single management cluster can support multiple pods of edge and compute clusters. The minimum number of hosts required in the management cluster is three, although four hosts are recommended for availability and performance. The management cluster can scale out as the number of edge and compute pods increases. With the exception of vsphere Data Protection, which stores all backups on NFS storage, all virtual machines utilize Virtual SAN storage. A single vcenter Server instance manages the resources in the management cluster. Additional vcenter Server instances manage edge and compute clusters. A Platform Services Controller is deployed for each vcenter Server instance. The Platform Services Controllers are joined to the same VMware vcenter Single Sign-On domain, enabling features such as Enhanced Linked Mode and cross vcenter Server VMware vsphere vmotion. For more information on the Platform Services Controller and new enhancements in vsphere 6.0, see the VMware vcenter Server 6.0 Deployment Guide and the What s New in vsphere 6.0 white papers. TECHNICAL WHITE PAPER / 7

8 VMware NSX Manager instances, one for each vcenter Server instance, are deployed into the management cluster. NSX for vsphere components, such as VMware NSX Controller instances, are also deployed for and in the management cluster. VMware NSX Controller instances for the edge and compute clusters are deployed into the edge cluster. External Access Internal Access Spine Leaf Virtual SAN Management External Storage Figure 3. Management Cluster TECHNICAL WHITE PAPER / 8

9 Edge Cluster The edge cluster simplifies the physical network configuration. It is used to deliver networking services to the compute cluster virtual machines. All external networking, including corporate and Internet, for user-workload virtual machines is accessed via the edge cluster. The minimum edge cluster size is three hosts, but it can scale depending on the volume of services required by the compute cluster virtual machines. External Access Internal Access Spine Leaf Virtual SAN Edge Figure 4. Edge Cluster TECHNICAL WHITE PAPER / 9

10 Compute Clusters The compute clusters are the simplest of the three types; they run user-workload virtual machines. Compute cluster networking is completely virtualized using NSX for vsphere. A single transport zone exists between all compute clusters and the edge cluster. Virtual switches are created for user-workload virtual machines. The minimum compute cluster size is 4 hosts; the maximum is 64 hosts. Additional compute clusters can be created until the maximum number of either hosts (1,000) or virtual machines (10,000) for vcenter Server is reached. Additional vcenter Server instances can be provisioned in the management cluster to facilitate more compute clusters. Spine Leaf Virtual SAN Virtual SAN Compute External Storage Figure 5. Compute Clusters TECHNICAL WHITE PAPER / 10

11 Physical Component Details Compute Table 3 lists the recommended minimum physical server configuration. COMPONENT SPECIFICATION CPU 24GHz Two 2GHz six-core CPUs (12 total cores) Memory 256GB ECC RAM SD 6GB SD card boot device HDD controller Virtual SAN certified controller* Flash 500GB Virtual SAN certified Flash device* HDD Two 1TB Virtual SAN certified HDDs* Network interface cards Two 10Gb network adapters Power supplies Redundant Fans Redundant Table 3. Minimum Physical Server Configuration * Virtual SAN certified devices can be found at For ease of management and to guarantee resource availability as the solution grows, all physical server hardware, regardless of cluster, utilizes the same configuration. TECHNICAL WHITE PAPER / 11

12 Storage The management cluster utilizes Virtual SAN in addition to NFS datastores. Virtual machines reside on Virtual SAN; vsphere Data Protection backups reside on NFS datastores. The edge cluster utilizes Virtual SAN, which serves the VMware NSX Controller instances for the edge and compute clusters as well as NSX Edge devices. The compute clusters utilize Virtual SAN, NFS, and VMware vsphere VMFS datastores. The size and number, if any, of datastores other than Virtual SAN depend on available capacity, redundancy requirements, and application I/O needs. Table 4 presents some guidelines for sizing storage. STORAGE CLASS IOPS (PER 100GB) MB/SEC (PER 1TB) REPLICATION DEDUPLICATION Gold Yes Yes* Silver No Yes* Bronze 25 2 No Yes* Table 4. Storage Sizing Guidelines * Deduplication is enabled only on storage systems that support this feature. Performance values are based on 100 percent random I/O with 70 percent read and 30 percent write rate at 8KB block size. Network Each rack contains a pair of multichassis link aggregation capable 10 Gigabit Ethernet (10GbE) top-of-rack switches. Each host has one 10GbE network adapter connected to each top-of-rack switch. The vsphere hosts utilize the VMware vsphere Distributed Switch configured with an 802.3ad Link Aggregation Control Protocol (LACP) group that services all port groups Q trunks are used for carrying a small number of VLANs for example, NSX for vsphere, management, storage, and vsphere vmotion traffic. The switch terminates and provides default gateway functionality for each respective VLAN; that is, it has a switch virtual interface (SVI) for each VLAN. Uplinks from the top-ofrack switches leaf layer to the spine layer are routed point-to-point links. VLAN trunking on the uplinks even for a single VLAN is not allowed. A dynamic routing protocol (OSPF, ISIS, or BGP) is configured between the top-of-rack and spine layer switches. Each top-of-rack switch advertises a small set of prefixes, typically one per VLAN or subnet that is present. In turn, it calculates equal cost paths to the prefixes received from other top-of-rack switches. TECHNICAL WHITE PAPER / 12

13 Spine OSPF Leaf (Top-of-Rack) LACP VMware ESXi Host Figure 6. ESXi Host Network Connectivity TECHNICAL WHITE PAPER / 13

14 Software-Defined Data Center Component Details In this section, we will define the VMware software components and their configuration in enabling this solution. COMPONENT NUMBER DEPLOYED DEPLOYED LOCATION CONNECTED NETWORK Platform Services Controller 2 Management cluster vsphere management VLAN vcenter Server 2 Management cluster vsphere management VLAN ESXi hosts Minimum of 1 1 (varies based on compute cluster requirements) 4 Management cluster 3 Edge cluster 4 Compute cluster ESXi management VLAN vsphere Data Protection 1 Management cluster vsphere management VLAN NSX Manager 2 Management cluster vsphere management VLAN VMware NSX Controller 6 3 Management cluster 3 Edge cluster vsphere management VLAN vrealize Operations Manager 4 (1 master, 1 master replica, 2 data nodes) Management cluster vrops Table 5. SDDC Component Details TECHNICAL WHITE PAPER / 14

15 vsphere Data Center Design The vsphere Enterprise Plus Edition is the core that enables the SDDC. All vsphere hosts are stateful installs that is, the ESXi hypervisor is installed to a local SD card. ATTRIBUTE SPECIFICATION ESXi version 6.0 U1 Number of hosts 11 Number of CPUs per host 2 Number of cores per CPU 6 Core speed Memory Number of network adapters 2GHz 256GB 2 x 10Gb Table 6. ESXi Host Details The solution uses VMware vsphere High Availability (vsphere HA) and VMware vsphere Distributed Resource Scheduler (vsphere DRS). vsphere HA is set to monitor both hosts and virtual machines. Its admission control policy utilizes a percentage of cluster resources reserved 25 percent in a four-node cluster guaranteeing sustainability with one node failure. To calculate the percentage, divide 100 by the number of hosts in a cluster; then, multiply the result by the number of hosts that can be eliminated while still guaranteeing resources for the virtual machines in the cluster. For example, an eight-host cluster for which sustainability for a two-host failure is the goal would be = 12.5 x 2 = 25 percent. vsphere DRS is set to fully automated mode. VLAN ID FUNCTION 14 VXLAN (management cluster) 24 VXLAN (edge cluster) 34 VXLAN (compute clusters) 970 ESXi management 980 vsphere vmotion 1020 IP storage (NFS) 1680 vsphere management 1701 External 3002 Virtual SAN Table 7. VLAN IDs and Functions TECHNICAL WHITE PAPER / 15

16 PORT GROUP VDS VL AN I D vds-mgmt-esxi vds-mgmt 970 vds-mgmt-ext vds-mgmt 1701 vds-mgmt-nfs vds-mgmt 1020 vds-mgmt-vmotion vds-mgmt 980 vds-mgmt-vsan vds-mgmt 3002 vds-mgmt-vsphere-management vds-mgmt 1680 VXLAN (VMware NSX autocreated) vds-mgmt 14 vds-edge-esxi vds-edge 970 vds-edge-ext vds-edge 1701 vds-edge-vmotion vds-edge 980 vds-edge-vsan vds-edge 3002 vds-edge-vsphere-management vds-edge 1680 VXLAN (VMware NSX autocreated) vds-edge 24 vds-comp-esxi vds-comp 970 vds-comp-nfs vds-comp 1020 vds-comp-vmotion vds-comp 980 vds-comp-vsan vds-comp 3002 VXLAN (VMware NSX autocreated) vds-comp 34 Table 8. Port Groups DATASTORE TYPE FUNCTION DS-VSAN-MGMT01 Virtual SAN Management cluster virtual machine datastore DS-NFS-MGMT01 NFS vsphere Data Protection backups DS-VSAN-EDGE01 Virtual SAN Edge cluster virtual machine datastore DS-VSAN-COMP01 Virtual SAN Gold-tier storage DS-NFS-COMP01 NFS Silver-tier storage DS-NFS-COMP02 NFS Silver-tier storage Table 9. Datastores TECHNICAL WHITE PAPER / 16

17 ATTRIBUTE SPECIFICATION vcenter version Quantity Appliance size 6.0 U1 (appliance) Four (two Platform Services Controllers, two vcenter Server instances) Small for management vcenter Server, large for compute vcenter Server Table 10. VMware vcenter Configuration ATTRIBUTE SPECIFICATION Data center object Enhanced Linked Mode WDC01 Automatic by joining same vcenter Single Sign-On domain Table 11. VMware vcenter Data Center Configuration vsphere Data Protection vsphere Data Protection is deployed in the management cluster and is responsible for backups and restores of the virtual machines residing in the management cluster. A backup policy was created for each application such as vcenter Server, vrealize Operations, and so on. Backup frequency and retention periods vary based on organizational requirements. Nightly backup of all virtual machines and databases is recommended. Figure 7. vsphere Data Protection TECHNICAL WHITE PAPER / 17

18 NSX for vsphere NSX for vsphere provides the virtual switches, routing, and load balancer services used to create the SDDC. All virtual machine traffic, excluding that on the vsphere management VLAN, is encapsulated using NSX for vsphere. All virtual machine to virtual machine, or east west, traffic is encapsulated and then routed between the virtual tunnel endpoints (VTEPs) of the host, where it is decapsulated and delivered to the virtual machine. When a request to or from the external network is serviced, it travels through the NSX Edge device in the edge cluster, which provides all north south routing that is, routing to and from external networks. NSX for vsphere has a one-to-one relationship with vcenter Server, so two NSX Manager instances are deployed, one for the management cluster vcenter Server instance and the other for the edge and compute cluster vcenter Server instance. These are both deployed in the management cluster. NSX for vsphere utilizes controller virtual machines to implement the network control plane. The VMware NSX Controller instances must be deployed in odd numbers to avoid a split-brain scenario. As such, three controllers per NSX for vsphere instance are deployed. The VMware NSX Controller instances for the management cluster are deployed into the management cluster itself; the instances for the edge and compute clusters are deployed into the edge cluster. The ESXi hosts must be prepared for NSX for vsphere. The following values are used: SPECIFICATION VALUE MTU 9000 Teaming mode LACP V2 VLAN 14 (management) 24 (edge) 34 (compute) Segment IDs Transport zones Management (management cluster) Compute (edge and compute clusters) Table 12. NSX for vsphere Values To enable external (north south) connectivity for the compute workloads, an NSX Edge router is deployed in HA mode. This NSX Edge instance is referred to as the provider edge. One interface is connected to the external network; another is connected to an NSX virtual switch, which is also connected to an NSX Edge router. The NSX Edge routers are configured with OSPF to facilitate the exchange of routing information. This enables the virtual machines on the NSX virtual switch to communicate with the external network and vice versa as long as firewall rules permit the communication. TECHNICAL WHITE PAPER / 18

19 External Network/Internet Spine Leaf External VLAN Virtual SAN Virtual SAN Compute Transport Zone VXLAN 5001 VM VM Edge Compute Figure 8. NSX for vsphere North South Routing TECHNICAL WHITE PAPER / 19

20 Monitoring Monitoring the performance, capacity, health, and logs in any environment is critical. Using vrealize Operations along with vrealize Log Insight is a unified management solution for performance management, capacity optimization, and real-time log analytics. Predictive analytics leverages both structured and unstructured data to enable proactive issue avoidance and faster problem resolution. The solution extends intelligent operations management beyond vsphere to include operating systems, physical servers, and storage and networking hardware. It is supported by a broad marketplace of extensions for third-party tools. vrealize Operations vrealize Operations provides operations dashboards, performance analytics, and capacity optimization capabilities needed to gain comprehensive visibility, proactively ensure service levels, and manage capacity in dynamic virtual and cloud environments. vrealize Operations is deployed as a virtual appliance and is distributed in the OVA format. In this architecture, vrealize Operations is deployed on an NSX virtual switch. Four vrealize Operations appliances are deployed. The first is configured as the master node, the next as the master replica, and the last two as data nodes. The four appliances access the vsphere management VLAN via the NSX Edge device configured with either static or dynamic routing. The NSX Edge device also load-balances the four virtual appliances on port 443, providing access to the vrealize Operations cluster via a single FQDN. External VLAN vsphere VLAN NSX vswitch /24 VM VM VM VM NSX Edge (Internal) (vsphere) (External) vrealize Operations Appliances Load-Balanced IP Figure 9. vrealize Operations TECHNICAL WHITE PAPER / 20

21 To ensure a complete picture of how the environment is running, vrealize Operations is configured to monitor the management, edge, and compute vcenter Server instances. Additionally, the NSX for vsphere content pack is installed and configured to provide insight into the virtualized networking environment.. Figure 10. vrealize Operations Dashboard vrealize Operations requires updates to the default monitoring settings for most organizations. For more information on how to customize vrealize Operations for a specific environment, see the vrealize Operations documentation. Conclusion The reference architecture presented in this paper describes the implementation of a software-defined data center (SDDC) that uses the latest VMware components as its foundation. Customers following this architecture can be confident that they will have the best possible supported configuration, one that is fully backed by the VMware Validated Design process. For a guided tutorial that shows step-by-step instructions for deploying this configuration, see TECHNICAL WHITE PAPER / 21

22 The VMware Validated Design Team Blaine Christian, Scott Faulkner, Phil Weiss, Christian Elsen, Nik Gibson, Randy Jones, William Lam, Nick Marshall, Paudie O Riordan, Kamu Wanguhu, Steven Ching, Michelle Gress, Christine Zak, Yu-Shen Ng, Bob Perugini, Justin King, Karthik Narayan, Sunny Bhatia, Mandar Dhamankar, Olga Efremov, David Gress, Kristopher Inglis, Rama Maram, Hari Krishna Meka, Arvind Patil, Venkat Rangarajan, Lakshmanan Shanmugam, Todor Spasov, Georgi Staykov, Antony Stefanov, Kevin Teng, Todor Todorov, Tuan Truong, Randy Tung, Shivaprasad Adampalli Venkateshappa, Lap Vong, Zhuangqian Zhang, and Mike Brown About the Author Mike Brown is a senior technical marketing architect in the Integrated Systems Business Unit. Mike s focus is on reference architectures for VMware vcloud Suite and the software-defined data center. He has multiple industry certifications, including VMware Certified Design Expert (VCDX), VMware Certified Advanced Professional Cloud, and VMware Certified Professional Network Virtualization. Follow Mike on the vsphere Blog and on TECHNICAL WHITE PAPER / 22

23 VMware, Inc Hillview Avenue Palo Alto CA USA Tel Fax Copyright 2015 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. Item No: VMW-RA-Sftwr-Def-Data-Cntr-USLET-101 Docsource: OIC-FP-1379