vsphere Basic Site Reference Architecture!

Transcription

1 A CookBook v1.1 March 12, 2014 vsphere Basic Site Reference Architecture Author(s) & Contributor(s) (IBM) Merlin Glynn, Robert Kellenberger Shane B. Mcelligott Daniel de Araujo Marco Prado Rodriguez (VMware) Coby Litvinsky Marcos Hernandez Geoff Wing 1 of 52

2 vsphere Basic Site Reference Architecture I Summary The core objective of this series of VMware@SoftLayer CookBooks is to enable vsphere administrators with key information to deploy VMware vsphere environments within SoftLayer. SoftLayer offers a very unique capability to VMware administrators to consume Bare Metal instances and network/storage/backup&recovery constructs from SoftLayer in a self service cloud construct manor. These constructs can be utilized to deploy fully functional vsphere implementations that can be architected to extend or replace on premises vsphere implementations (VMware@Home) and enable VMware administrators to realize Hybrid Cloud characteristics rapidly, and in a cost effective manor by deploying into SoftLayer s Enterprise grade Global Cloud (VMware@SoftLayer). This is a key differentiator from other cloud providers like Amazon Web Services, in that vsphere workloads and catalogs can be provisioned onto VMware vsphere environments within SoftLayers Global Cloud DataCenters, without modification to VMware VMs or guests. A common vsphere hypervisor and management/orchestration platform makes this possible. vsphere implementations in SoftLayer also enable utilization of other components of the VMware vcloud Suite such as vcloud Automation Center, vcenter Operations Management Suite, vsan, vcloud Network & Security, Site Recovery Manager, vcenter Orchestrator, and NSX. This document will focus on setting up an initial Single Site, vsphere 5.5 implementation within SoftLayer, in a self service manor. It will provide fundamental information on the following key topics: II SoftLayer Networking & VPN Management Access, a High Level Overview III VMware vsphere ESX 4.x & 5.x Installation & Licensing Options IV vsphere vcenter 5.5 Installation & Licensing Options V Basic Site SoftLayer Private/Public vsphere Network Design VI Basic Site SoftLayer vsphere Storage design & Simple BURA (Backup Recovery & Archival) VII Basic Site Recipe (How To Deploy) Note: This document is intended for experienced vsphere Administrators. Some topics covered will consider that the reader has basic deployment skills to Install and Configure vsphere & vcenter 4.x & 5.x. " Note: This document is NOT intended to provide enablement on basic Operating System Tasks within VM Guest Operating Systems." 2 of 52

3 II SoftLayer Networking & VPN Management Access High Level Overview Before deploying vsphere within SoftLayer, it is important to understand basic SoftLayer networking constructs. This section will highlight the basic network architecture components you will need to understand to deploy vsphere on servers at SoftLayer. figure 1" Figure 1 (above) depicts SoftLayer s hi-level network architecture. There are 3 key components to understand from a networking perspective, these are (1)Private and (2)Public Networks, and (3)IP Subnets. Private & Public networks are implemented as 802.1q VLANS. By default, when a new tenant provisions a new server into a SoftLayer datacenter with no other pre-existing tenant deployments, qty 1 Public Network & qty 1 Private Network is created, and qty 1 IP Subnet is associated with each. Successive deployments leverage these same VLANS by default. Characteristics of each network type include(s) ( network): Public Network VLANs" Can Have One or More IPv4/IPv6 subnets provisioned from SoftLayer & routed to them. Are by default not protected (No Firewall Rules) unless Firewall/Gateway services ordered. SoftLayer Provisioned IP subnets on Public VLANs can be routed through a Vyatta Gateway security appliance(s) or protected by SoftLayer hardware FireWall services. 3 of 52

4 Private Network VLANs" Can Have One or More IPv4/IPv6 subnets provisioned from SoftLayer routed internally to them. Subnets provisioned Are by default NOT routable publicly/outbound from the datacenter, but are accessible via SoftLayer remote management VPN access. Can be accessed for management via vpn.softlayer.com(pptp/ssl). Can be enabled to L3 route between multiple private networks, even across SoftLayer data center boundaries.( SoftLayer Provisioned IP subnets on Private VLANs can be routed through a Vyatta Gateway security appliance(s) for added security. vsphere environments are recommended to have at least 3 Private Networks provisioned & 1 Public Network Provisioned. This will Allow for the following Layer 2 segment traffic boundaries: VLAN1 = Management VMK & VM IP Traffic VLAN2 = vmotion/ft/vsan Traffic VLAN3 = VM Access Layer Traffic *Optionally a fourth VLAN should be added if VXLAN will be utilized (VLAN4 = VXLAN SDN Transport Traffic). VXLAN & NSX will be covered in detail in another VMware@SoftLayer CookBook. For more details on VXLAN & NSX, please reference VMware@SoftLayer CookBook: NSX@SoftLayer. 4 of 52

5 Subnets are another important SoftLayer network object to understand. They are utilized for routing & IP address management from SoftLayer. They are address blocks that are routed to specific VLANs. They can also be routed to specific IP addresses by SoftLayer (Static) to a tenant deployed routing/security appliance (Vyatta, NSX Edge gateway, etc ) They are provisioned in a few types: Network Type Primary vsphere Use Case Auto Assigned to CCIs & Physical Servers Nics by SL Can Route Over Public by Default Can Route Over Public w/ Gateway (NAT) for added security and control vlan Spanning (L3 Routing) For multi-site routing Private Primary Private Portable Public Primary Public Portable vmk0 Management Virtual NIC IPs & IPMI Access VM private RFC IP addresses & other vmk nics Assigned to public facing Physical NICs, Not Leveraged by ESX by default Public IPs that can be assigned to VMs, Virtual Edge Gateways (NSX/VCNS) YES NO YES YES NO NO YES YES YES YES YES NO NO YES YES NO Public Static Forward public IP range to specific other public IP Address for routing (NSX/ VCNS). NO YES YES NO * It is possible for customers to bring their own IP space in SoftLayer, but this scenario requires use of Customer Provided Network links to a specific SL POP location and or NAT/IPSEC 5 of 52

6 Tunneling & Routing over private networks. For more details on BYOIP, please reference CookBook: Advanced Connectivity. * With respect to subnets & VLANs, it is important to note that Private & Public networks can be provisioned without an IP subnet via support tickets to SoftLayer, thus requiring the tenant to provision VMs or physical boxes to to handle inter VLAN routing and North/South traffic patterns if so desired (Example Use case vsan/iscsi subnet that only require Layer 2). This Cookbook will present a Basic vsphere scenario and a guide as to which SoftLayer Network Options should be selected to address a Basic Single Site vsphere implementation use case. VPN Management Access vsphere Administrators can leverage the SoftLayer VPN portal access method to initiate management access to one or more Private Networks/VLANS. This is depicted in Figure 2. Access to Private Networks is provided via the VPN Portal. The VPN Supports SSL or PPTP based tunneling mechanisms. figure 2" Access to various subnets can be restricted by using the SoftLayer Portal and can be navigated to via Private Networks -> VPN, then editing the specific user you wish to limit VPN access (Figure 2). *Note:If you choose to route Networks to a Vyatta Gateway Appliance, you will be required to set appropriate Firewall rules for SoftLayers VPN portal & management layer to still provide access to your Networks. this concept is covered in detail here ( knowledgelayer.softlayer.com/faq/what-ip-ranges-do-i-allow-through-firewall). 6 of 52

7 III VMware vsphere ESX 4.x & 5.x Installation & Licensing This section will cover ESX Installation, it will provide basic SoftLayer UI direction and guidance for SoftLayer. Licensing When provisioning ESX from SoftLayer You will have 2 basic choices related to Installation & licensing. A. VMware from SoftLayer = You may provision an ESX 4.0/4.1/ or 5.1 server from SoftLayer. This Operating System Type is available after you proceed to the sales order form (Sales -> Add Server) and select your server configuration. Choosing this method will deploy a vsphere server with Enterprise Plus Licensing and the following characteristics: figure 3 ESX version selected will be pre-installed. A minimum of 2 vmnics will be deployed, 1 on a Public Network (no PortGroups or VMK IPs on the Public Interface), the second on a Private Network. It is possible & recommended to order dual uplinks for Public & Private Networks, this will place 2 physical nics in each Public & Private Network. (More than 4 Physical Nics are available but are not ordered self service via the portal. Contact [email protected] or create a support ticket for assistance to select a capable server). 2 IP addresses from the Primary Subnet of Private Network will be provisioned (1 for IPMI & The other for vmk0 Management Access), 1 Public IP address will also be assigned in the portal but not in ESX. A vsphere local user (vmadmin) on each host will be created, this user MUST remain enabled and password synchronized with the SoftLayer Portal server control panel. This user is inject by SoftLayer to collect polling information such as # of Running VMs & Used Memory. This information is collected and applied against VMware VSPP licensing to derive your monthly billing. This is not required If you bring your own license. How-To Deploy vsphere ESX 5.5 = After servers are deployed, VMware Update Manager (VUM) or Out Of Band Remote Access (Covered later in this section) can be utilized to upgrade from 5.1 -> of 52

8 B. VMware from BYOL (Bring Your Own License) = You may provision ESX directly via Remote Console and Virtual media access. This method would leverage your own licenses from VMware and bill only the physical components of an ESX server monthly from Softlayer. Choosing this method, you will deploy a Free and Open Source Software (FOSS) operating system on the server (at the time of this writing, NO OS is not available from the SoftLayer portal when ordering a server). Choosing this method will deploy a physical server with the following characteristics: Centos (Or Chosen FOSS) will be pre-installed. A minimum of 2 nics will be deployed, 1 on a Public Network, the second on a Private Network. It is possible & recommended to order dual uplinks for Public & Private Networks, this will place 2 physical nics in each Public & Private Network. (More than 4 Physical Nics are available but are not ordered self service by the portal. Contact [email protected] or create a support ticket for assistance to select a capable server). 2 IP addresses from the Private Network will be provisioned (1 for IPMI & The other for eth0 & eth2 Management Access), 1 Public IP address will also be assigned to eth1 & eth3. How-To Deploy vsphere ESX 5.5 = Out Of Band Remote Access (Covered later in this section) can be utilized to install. vsphere Autodeply & Host Profiles can also be utilized to speed deployment of multiple ESX hosts. figure 4" 8 of 52

9 Ordering Your Server(s) Theses are recommendations of Tasks to perform when ordering your ESX Servers: 1. Order Your Private Network(s) before you order your servers. These can be ordered via submitting a Standard Support ticket in the SoftLayer Support->Tickets. You should select Private Network Question as the subject and request your VLANs (Figure 5). You can self provision your first Private Network by simply ordering a Private Subnet via the portal, but additional VLANs require a Ticket submission. Repeat this process for at least 1 Public Network. The recipe section of this document will cover this concept in more detail. figure 5 Having your Public & Private Networks already in place simplifies placement when you order your server as they will be available in a drop down list when you provision your new server(s). This will allow you to ensure your servers are provisioned with access to appropriate VLANs. 2. When Ordering your servers from the SoftLayer Sales->Add Server, consider your CPU & Memory requirements. Dual uplinks (4 Nics) for HA architecture are sugested. If your design will require more vmnics, consider selecting a server that will allow > 4 NICs. These are usually the Mass Storage & Redundant power models, but can be selected definitively by contacting [email protected] for assistance. figure 6 9 of 52

10 3. After your servers are ordered, If you will be utilizing the Bare Metal KVM & Remote media method of installing ESX, It is advised that you file a ticket with SoftLayer support to elevate your IPMI access from Operator to Administrator. The default Operator role is not entitled with Virtual Storage mounting. 4. Multiple Private Networks Can be trunked to the ESX servers, allowing for the Virtual Switches to apply VLAN Tagging at the Port Group layer for layer 2 isolation. This process is not accomplished via the portal, but is accomplished via a SoftLayer support ticket. 10 of 52

11 Installation Methods A. ESX Bare Metal Install via KVM & Virtual media" This method of deploying ESX from Installation media is similar across all versions and should be utilized in BYOL scenarios. Requirements A VM instance that has access to the Private Network in SoftLayer (i.e., a CCI installed with Windows or Linux, with a Java enabled browser, accessible via vpn.softlayer.com). This CCI must be on the same Private Network that the server s IPMI addresses are located. A copy of VMware ESXi VIM Installer ISO is also required. Steps 1. Upload the VMware ISO to the CCI specified in requirements (The CCIs should have public web access). 2. Gather the IPMI address and login information from the SoftLayer Hardware -> Control -> [Server] -> Hardware -> Network. Scroll down to Network and click on the IPMI link for mgmt0 figure 7 figure 8 On the next web page you will see the IPMI address and the login details for the server you are installing. 3. RDP/Remote X into to the CCI that is storing the ESXi image. 4. In the RDP session, open a web browser and enter the IPMI address you collected from the previous step. Login to the IPMI console with the credentials also found in the same step (typically root). 5. After login, take note of your IPMI user access level. It should be Administrator. If it is set at Operator, you may experience problems mounting your remote storage, and should file a support ticket with Softlayer to elevate your IPMI credentials if you are unable to mount media. 11 of 52

12 6. On the home login page, select Remote Control->Console Redirection then click on the Launch Console button. figure 9 7. A KVM viewer should now be present. On the viewer, select Virtual Media->Virtual Storage. Your operating system and java version may require you to allow access to launch the Java based viewer. figure Within the virtual storage window, select the CDROM&ISO tab. Within the tab, select ISO file, then click on the open image button. Navigate to the ESXi ISO, click OK, then click on the Plug in button. Click OK after the ISO has been plugged into the session. figure of 52

13 9. Go back to the IPMI webpage and reboot the server. Remote->Control, select Reset Server and click Perform Action. figure Install ESXi as you normally would. 11. Once you've completed the install, make sure to Plug Out the ISO so you can reboot the server. figure Reboot the server. 13 of 52

14 13. After the server reboots, configure the IP address of the server. Gather the details from the Hardware->Configuration page on the SoftLayer site. figure At this point, you should now be able to add the newly installed server to a vcenter Instance. 14 of 52

15 B. ESX upgrade via VMware Update Manager" This method of of upgrade requires that VMware Update Manager (VUM) be deployed in addition to VMware vcenter. vcenter & VUM may be deployed in a CCI that is connected to the same Private Network as vsphere Management VMKs. Alternatively, vcenter & VUM can be deployed as VMs / OVF Appliances directly onto an ESX Management Cluster as described in the Basic Site Architecture in this document. VUM can be applied against SoftLayer provisioned ESX or BYOL ESX. C. ESX Install via Auto Deploy & Host profiles" This method allows for rapid and policy driven deployments of ESX. The Autodeploy infrastructure components (which are all virtual) can be deployed within SoftLayer and leveraged to provision ESX at scale. language=en_us&cmd=displaykc&externalid= " 15 of 52

16 IV Planning: vsphere vcenter 5.5 Installation & Licensing Options vcenter is a key component in any vsphere deployment, enabling services like DRS, HA, and distributed virtual switching. It is also the key component that many other VMware vcloud Suite products reply upon (vcac, vcd, VCOPS, etc ). There are 2 primary choices for deploying vcenter in SoftLayer s Global Cloud Datacenters: 1. Order vcenter as a SoftLayer Server or CCI: SoftLayer provides a Windows Based vcenter 5.1 Standard installation that can be provisioned as part of a new server order or a CCI (Cloud Computing Instance). A CCI is a XEN based VM provided by SoftLayer s Multi Tenant VM infrastructure. It can have its virtual interfaces wired to the same VLANs as your vsphere infrastructure s management access. This method is desirable if you will be deploying ESX and licensing ESX from SoftLayer. Items to be aware of: Can be upgraded to language=en_us&cmd=displaykc&externalid= Is licensed via SoftLayers VSPP agreement with VMware, billed to tenant monthly. Can be selected as an OS specific Add-On for a bare metal server instance or CCI Must be Windows based. figure of 52

17 2. Deploy vcenter as a VMware VM in a vsphere Management Cluster: It is a VMware best practice when deploying vsphere to dedicate a cluster of hosts to provide virtual infrastructure for various VMware management VMs. vcenter is one of many VMware management VMs in the vcloud Suite that can be deployed as an OVF Linux based appliance (Including SSO) or on a Windows Instance VM. This method has many advantages: vcenter 5.5 can be deployed as base install instead of upgrade. Is licensed via Customer BYOL. vsphere VM Does not require Public Uplinks, allowing for easier security zoning. Can be deployed on Windows platform or via Linux based OVF Appliance. In addition, deploying an ESX management cluster allows for easier deployment of other VMware vcloud Suite Products that are OVF based (VCOPS, VCAC, SRM Host Based Replication, VCNS, NSX, vcc, VCO, and VIN). These components are delivered in OVF format and intended to be deployed directly on ESX hosts. VMware has stated a direction to develop and support vcenter in an appliance form factor, thus making a vsphere Management cluster a good design choice when deploying in SoftLayer. figure 16 Installing vcenter can be accomplished by: Windows Based: Push a windows ISO or OVF Template to a local datastore on a deployed ESX host. This can be up/down-loaded either via SoftLayers Management VPN or by deploying a Windows or Linux utility CCI to download and push files to the VMware infrastructure. A Windows instance can then be deployed and vcenter installed. OVF Appliance: Pushing the OVF to a Windows or Linux CCI and leveraging the traditional vsphere C Client (Windows) or OVFTOOL & RVC (Windows or Linux) to deploy the vcenter Virtual Appliance (VCVA). 17 of 52

18 V Basic Site Reference Architecture : SoftLayer Private/Public vsphere Network design w/ Vyatta & VPN Access Layer SoftLayers s Multi datacenter VLAN architecture does offer some compelling options for vsphere design. With proper application, most on Premises <-> SoftLayer & SoftLayer <-> SoftLayer (multi Site) scenarios can be accommodated. This section will focus on a single site deploy and provide some basic VPN Interconnect options. For more details on Multi Site deployments, please reference VMware@SoftLayer CookBook: Advanced Connectivity. Logical Network Overlay VMNICs" Figure 16 depicts a basic vsphere Site with a Management Cluster and a Capacity Cluster. Each host across both clusters has been deployed as Dual Public & Private Networks uplinks. This provides links in the following manor: figure of 52

19 - vmnic0(eth0) & vmnic2(eth2) are connected to Primary Private Network VLAN Provision time - vmnic1(eth1) & vmnic2(eth3) are connected to Primary Public Network VLAN Provision time (These Uplinks Can be Disabled) If vsphere 4.x/5.1 was provisioned from SoftLayer s catalog, your vmk Mgmt IPs will have been pre-assigned from the Primary Private Subnet that was attached to the chosen VLAN you selected when you ordered. vmnics 0 & 1 will be set in an active passive configuration and vmk0 active. vmnics 1 & 3 will be physically connected to the Public VLAN chosen during deploy, but no vmks or VM PortGroups will be assigned. If vsphere 5.x was installed manually, you will decide how the vmnics will be configured in ESX, but the physical interfaces will still be set similarly with the Primary Private & Public networks you chose during provisioning. At this time, SoftLayer does NOT support moving the interfaces (eth1 & eth3) on the Public Network Ports to another Private Network, however they can be disabled. There are Specialty Servers: Private Network server order choices that allow for servers to have no Public Network connects (public disconnected), but these servers are limited in size and expansion options and are not recommended for selection as ESX hosts at this time. To secure workloads, it is recommended to not utilize vmnics 1 & 3 (eth1&eth3) connected to Public Networks and remove them from or not connect them to any vswitches/dvswitches, unless they are to be utilized as gateway ESX hosts in a SDN deployment with virtual gateway appliances or public access is desired. This scenario will be discussed VMware@SoftLayer CookBook: Advanced Connectivity document for gateway services to secure North/South traffic, as well as provide East/West traffic overlays. Additionally, the public switch ports may be disabled by navigating to the SoftLayer Public Network -> Port Control. This document will present a reference architecture where each host has 4 x 1 GBps vmnics (more physical nics & single uplink 10GB nics can be ordered assuming the physical server ordered has PCI expansion bays for the appropriate or additional interface cards. In order to ensure your servers can expand to > 4 physical nics, contact [email protected] for assistance BEFORE ordering your ESX servers). Additional nic pairs beyond the first 4 can be placed on Private Networks other than the Private Network assigned to vmnic0 & vmnic2 (eth0 & eth2), but must be ordered via contacting [email protected] or opening a support ticket. 19 of 52

20 VLANs & Public/Private Networks" As mentioned in Section II of this document (SoftLayer Networking & VPN Management Access High Level Overview), multiple VLANs can be provisioned and trunked to ESX host interfaces. In Figure 16, 4 x SoftLayer Networks have been provisioned: Public Network (dal0#.fcr01a.4000) VLAN=4000 PUBLIC - Primary Subnet w/ 8 IP addresses (Used For CCI s & Servers, Auto Provisioned by SL) - Portable Subnet w/ 8 IP addresses (Used For VMs requiring Public Interface, SDN Gateway Appliances, or DNAT IPs) Private Network (dal0#.bcr01a.1101) VLAN=1101 PRIVATE - MGMT - Primary Subnet w/ 64 IP Addresses (Used For CCI s & Servers, Auto Provisioned by SL) (These IP Addresses are utilized for vmk0 Management IP ESX Host Access) - Portable Subnet w/ 32 IP Addresses (Used For Management VMs ex vcenter) Private Network (dal0#.bcr01a.1102) VLAN=1102 PRIVATE - STORAGE/vMOTION/FT - Primary Subnet w/ 32 IP Addresses (Used For CCI s & Servers, Auto Provisioned by SL) (These IP Addresses should be reserved for additional servers or CCIs as read ) - Portable Subnet w/ 32 IP Addresses (Used For Management ESX Host VMKs providing iscsi/vsan/vmotion/ft services) Private Network (dal0#.bcr01a.1103) VLAN=1103 PRIVATE - VMAccess - Primary Subnet w/ 32 IP Addresses (Used For CCI s & Servers, Auto Provisioned by SL) (These IP Addresses should be reserved for additional servers or CCIs as read ) - Portable Subnet w/ 32 IP Addresses (Used For Private RFC VM traffic that is to be protected/snat via Vyatta gateway appliance, similar to a DMZ zone) When the ESX hosts are provisioned a Primary VLAN is selected for private uplinks, for example VLAN1101. This VLAN is the default VLAN on the physical switch ports and therefore does not require traffic tagging. In order to trunk the remaining VLANs (1102 & 1103), a support ticket must be filed with SoftLayer. After completion, the VLANS will be trunked to the backend switch (BCS) ports. In order to utilize these VLANs, traffic MUST be tagged. Utilizing VLAN tagging, multiple layer broadcast domains can traverse a single pair of private network interface uplinks. 20 of 52

21 Gateway Services In Figure 16, a Vyatta Gateway Appliance is introduced. This appliance will provide Layer 3 Firewall, VPN, and NAT services. It will allow securing traffic between multiple private networks, public and private networks, and multiple public networks. In this Basic Single site reference architecture it will be utilized to: Secure Inbound Internet Traffic SNAT Outbound Traffic from vsphere VM Management traffic & VM Access traffic Segment vsphere VM Management traffic from VM Access traffic Provide Remote VPN access to vsphere VM Management Traffic Filter SoftLayer OOB Management VPN Access When a Vyatta is deployed, 2 additional networks (1 Private & 1 Public) are provisioned. These VLANS (represented as VLAN 10 & VLAN 4001) are intended to be gateway VLANS. This concept allows multiple network boundaries to be filtered through the Vyatta. By default each subnet deployed attached to a Public or Private network has a gateway that handles routing via SoftLayer s normal routing scheme. These gateways can be routed to the Vyatta. figure of 52

22 Figure 17 depicts the concept. A complete VLAN is routed to the appropriate gateway VLAN, including all IP Subnets associated with that VLAN. This is accomplished via the SoftLayer Private Network -> Gateway Appliances. The Recipe section of this document will have specific how-to details. When the routing action is performed, the Private Network VLAN is trunked on the uplinks of the Vyatta appliance and the SoftLayer provided default gateways for each subnet are disabled. After routing a VLAN to a Vyatta Gateway Appliance, the subnets you wish to allow access to, must have their gateway interfaces configured as vifs on the Vyatta. A Vyatta uses linux bonds on its paired ethernet interfaces (assuming you provision a dual uplink instance). A bond leverages vifs to tag traffic and participate in specific VLANs. In the basic site reference architecture, Private VLANs 1 & 3 as Well as Public VLAN 4000 are forwarded to the Vyatta gateway appliances gateway VLANs. The Vyatta appliance must then be configured to provide the default gateway IP addresses for each forwarded VLAN & IP subnet, then appropriate Firewall rules applied to allow traffic to traverse. The Vyatta Gateway can then be leveraged to provide DNAT/SNAT, VPN access for either remote users or connecting On Premisses <-> SoftLayer with Site to Site networks, even Layer 2 bridging. In this Basic Single Site reference architecture, site to site VPNs will not be covered. For more details on Multi Site deployments, please reference VMware@SoftLayer CookBook: Advanced Connectivity. 22 of 52

23 VI Basic Site Reference Architecture : Basic Site SoftLayer vsphere Storage design & Simple BURA SoftLayer Provides many storage options suitable for vsphere deployments for ESX usage. vsphere 4.x & 5.x VMs are stored on NFS exports or VMFS filesystems on top of block storage. vsan is another vsphere 5.5 storage option that as of the time of this writing, is in public beta. vsan leverages local disks on ESX hosts and creates a virtual aggregate datastore name space across assigned local disks from participating ESX 5.5 hosts. vsan requires at least 1 SSD physical drive in each ESX host. Shared storage & vsan are critical design components for vsphere features such as vmotion/storage vmotion, HA, and Fault Tolerance. NFS (NAS)" SoftLayer can provide multiple options to meet the requisite vsphere shared storage types: SoftLayer s NAS Storage service only provides CIFS/FTP based access as of the time of this writing. This may be useful for ISO storage or other common guest data store, but is not suitable for ESX vmdk storage. Quantastore: Can be self service provisioned from SL portal, and is a self configured storage (iscsi/nfs) server which can connect its uplinks directly on a SoftLayer VLAN (Private Network). This allows ESX hosts to have connected/trunked non ip routed access directly to a vmk for IP based storage. The key benefit is that you can use vsphere Software iscsi MPIO, and have more control over your storage configuration and hardware. As of this writing the only additional options for adding NFS storage are to contact SoftLayer engineering to bring your on premises virtual storage appliances (if supported by SoftLayer) or to deploy SoftLayer physical servers/ccis, attach storage, and export that storage as NFS with a self provisioned OS/Tool (OpenFiler, Linux.). Block VMFS (iscsi)" SoftLayer does offer iscsi storage in 3 basic formats: - iscsi Storage: Can be ordered & added when provisioning a physical server or by navigating to This storage is actually provisioned as specific Equalogic Targets on a SoftLayer Private internal service network. These iscsi targets are not made available directly on tenant Private Networks. This requires routing from a vmk ip and using vsphere teaming for HA connectivity. VMware support does not recommend this option (iscsi routing) since it requires specific physical switch security settings and is generally limited to fail over. - Quantastore: Can be self service provisioned from SL portal, and is a self configured storage (iscsi/nfs) server which can connect its uplinks directly on a SoftLayer VLAN (Private Network). This allows ESX hosts to have connected/trunked non ip routed access directly to a vmk for IP based storage. The key benefit is that you can use vsphere Software iscsi MPIO, and have more control over your storage configuration and hardware. Quantastore also provides many other capabilities such as replication of 52

24 - Physical Server/CCI: Deploy SoftLayer physical server(s)/cci, attach storage, and export that storage as iscsi with a self provisioned OS/Tool (Windows iscsi Services, OpenFiler, IET ). - It Is possible to bring tenant virtual storage appliances into a SoftLayer datacenter with SoftLayer Engineering approval. Block vsan" SoftLayer does offer SSD drives as an option when ordering physical servers for ESX deployments. It is required to provision at least 1 SSD drive per each ESX host that will provide vsan storage. Please refer to VMware_Virtual_SAN_Datasheet.pdf for more information to leverage SoftLayer. Logical Storage Overlay figure 18" Shown in Figure 18 is a Basic Single site reference architecture utilizing a QuantaStore server. It has been deployed with appropriate capacity and spindles to meet expected IO requirements of the workloads to be run in the site. The QuantaStore is connected directly to VLAN2(1102), allowing iscsi MPIO access for a proper ESX MPIO iscsi load balanced configuration. iscsi targets are created for both the management and capacity cluster. 24 of 52

25 Simple Backup Archival & Recovery (BURA) SoftLayer some basic options that provide guest agent based &/or VDP based backup/recover approaches : EVault Backup " Simple Windows- or web-based graphical user interface to initiate all backups and restores, and to configure all backup and automation options. Administrators can set backups to follow an hourly, daily, weekly or custom schedule, and to target full systems, specific directories or even individual files. Individual software agents are available that enable EVault to target more specific data in Exchange, SQLServer, Sharepoint, MSClusters and Oracle servers. Evault is a backup service that is billed per volume of data stored. It does require agent access for VMware environments and is not VADP enabled. For more details on Multi Site deployments, please reference VMware on SoftLayer CookBook: BCDR. Idera Server Backup" Provides high-performance disk-to-disk server backup featuring a central management and data repository. Protects data at block level, and unique disk blocks on the server are stored only once across all recovery points, increasing storage efficiency. Reduces backup windows from hours to minutes Easy-to-use, Web-based interface Multi-platform support for physical and virtual servers Maximum data protection for MySQL databases Supported OSs: CentOS, Debian, RedHat, Ubuntu Windows Server 2008 (64-bit) Windows Server 2008 R2 (64-bit) Windows Server 2012 (64bit) To order, select IDERA Server Backup as an OS-Specific Addon, and add Backup Agent Packs in any combination as needed for the number of servers or databases that will be backed up. IDERA is a backup service implementation that is licensed & supported through SoftLayer but is administered by the tenant. It does require agent access for VMware environments and is not VADP enabled. For more details on Multi Site deployments, please reference VMware on SoftLayer CookBook: BCDR. Softlayer Swift Storage w/ VDPA & Twinstrata ( Softlayer s Object level storage can also be utilized with VMware VDP & Twinstrata. This and many other BURA options will be covered in more detail in VMware on SoftLayer CookBook: BCDR. 25 of 52

26 VII Recipe/HowTo Build a Basic Single Site Reference Architecture: This recipe assumes you have a functional SoftLayer Account and no servers yet provisioned." 1. Order required VLANs & Subnets Order VLANs Utilize the SoftLayer Follow Link to Support -> Tickets -> Add Ticket -> Create Standard Support Ticket" Subject = Private Networks Question Title = Order VLAN Ticket Contents = Please create 3 Private Networks. Associate 1 x /26 (64 Addresses) Primary Private Subnet with the first VLAN & 2 x /27 (32 Addresses) Primary Private Subnets, one for each of the 2 remaining Private Networks/ VLANs. Please Create These in DataCenter X (example Dal01) for initial use with 5 Hosts, 1 QuantaStor & 1 Vyatta. Follow Link to Support -> Tickets -> Add Ticket -> Create Standard Support Ticket" Subject = Public Networks Question Title = Order VLAN Ticket Contents = Please create 1 Public Network. Associate 1 x /29 (8 Addresses) Primary Private Subnet with the VLAN. Please Create this VLAN in DataCenter X (example Dal01, same as above). After the Networks Are provisioned, make note of the VLAN numbers and assign to logical vsphere networks (please note your Network Names & VLANs will likely differ from those shown): Network VLAN ID vsphere VLAN Usage Primary SubNet Mask Subnet Reserved For SL Private - dal06.bcr01a VM Traffic - Management + Host Management vmk0 /26 Y Private - dal06.bcr01a vmotion/ft/storage /27 Y Private - dal06.bcr01a VM Traffic - Access Network /27 Y Public - dal06.fcr01a Public IPs for VPN,SNAT, & DNAT /29 Y 26 of 52

27 Order Subnets Continue only if the VLANs have been provisioned completely these subnets will be utilized for VM/VMK traffic and SNAT/DNAT from Gateway Utilize the SoftLayer Follow Link to Sales -> Add IP Addresses" Subnet Type Subnet Size Bound VLAN VM/ESX Host Usage Portable - Private /27 32 Addresses 1101 Mgmt VMs Portable - Private /27 32 Addresses 1102 vmk3-vmk4 for iscsi Portable - Private /27 32 Addresses 1103 VM Access Network Portable - Public /29 8 Addresses 4000 Public IPs for VPN,SNAT, & DNAT 2. Order 2 x Management & 3 x Capacity ESX servers Continue only if the VLANs have been provisioned completely" Utilize the SoftLayer Follow Link to Sales -> Add Server" If you cannot find a configuration option available from the portal that meets your needs (example: Number&Speed of Physical Nics, Memory FootPrint, etc..) please contact [email protected] for assistance ordering your servers BEFORE placing your order." Select Appropriate Sized Servers. Refer to VMware Support publications for minimum requirements. DEB8086A-306B-4239-BF76-E FC.html" language=en_us&cmd=displaykc&externalid= " The IBM Cloud Innovation Lab Chose the following for the reference architecture test environment. Please note that having dual public and private uplinks is a requirement for this reference architecture. Please confirm the datacenter where you created the VLANs has this capability." " Management Cluster (Qty 2)" " Server Configuration: Dual Processor Multi-Core Server " Server Configuration: 32 GB Memory each" " Software: OS = Centos 6, None for remainder" " Storage: 1 x 500 GB SATA, None for remainder " Miscellaneous Software: None for complete section" " Networking: " 5000GB Public Bandwidth 1Gbps Dual Public & Private Networks, None for Remainder Bind to VLAN 1101(Private Uplink pair) & VLAN 4000 (Public Uplink Pair) " Monitoring & Response: Default Settings " Business Continuance: None 27 of 52

28 " Capacity Cluster (Qty 3)" " Server Configuration: Dual Processor Multi-Core Server " Server Configuration: 64 GB Memory each" " Software: OS = Centos 6, None for remainder" " Storage: 1 x 100 GB SSD + 2 x 500 GB SATA, None for remainder " Miscellaneous Software: None for complete section" " Networking: " 5000GB Public Bandwidth 1Gbps Dual Public & Private Networks, None for Remainder Bind to VLAN 1101(Private Uplink pair) & VLAN 4000 (Public Uplink Pair) " Monitoring & Response: Default Settings " Business Continuance: None" 3. Order QuantaStor Server Continue only if the VLANs have been provisioned completely" Utilize the SoftLayer Follow Link to Sales -> Add Server" If you cannot find a server configuration that presents OSNEXUS/QunataStor as an Operating System Selection, please contact [email protected] for assistance in selecting an appropriate server. Select Appropriate Sized Server. You will need to choose the correct physical disk spindle selection to meet your IO needs. The IBM Cloud Innovation Lab Chose the following for the reference architecture test environment:" " QuantaStor Storage Server (Qty1)" " Server Configuration: Dual Processor Multi-Core Server " Server Configuration: 24 GB Memory" " Software: OS = OSNexus/QuantaStor 3 SM (16TB)" " Storage: 10 x 500 GB SATA, None for remainder (2diskRAID1 Part & 8diskRAID6 Part) " Miscellaneous Software: None for complete section" " Networking: " 0GB Public Bandwidth 1Gbps Dual Private Network, None for Remainder Bind to VLAN 1102(Private Uplink pair) " Monitoring & Response: Default Settings " Additional Products & Services: None 28 of 52

29 4. Order Vyatta Gateway Appliance Continue only if the VLANs have been provisioned completely" Utilize the SoftLayer Follow Link to Sales -> Add Gateway Appliance" " " " Server Configuration: Dual Processor Multi-Core Server " Server Configuration: 4 GB Memory" " Software: OS = Vvatta 6.x Subscription Edition (64 bit)" " Storage: 1 x 500 GB SATA, None for remainder (2diskRAID1 Part & 8diskRAID6 Part) " Miscellaneous Software: None for complete section" " Networking: " 5000GB Public Bandwidth 1Gbps Dual Public & Private Networks, None for Remainder Backend VLAN=1001,Frontend VLAN=4000 " Monitoring & Response: Default Settings " 5. Order Utility CCI Continue only if the VLANs have been provisioned completely Utilize the SoftLayer Follow Link to Sales -> Add Monthly Computing Instance" " Server Configuration: 1 x 2.0 GHz Core " Server Configuration: 4 GB Memory" " Software: OS = Windows Server 2008 R2, None for remainder" " Storage: 25GB, None for remainder " Miscellaneous Software: None for complete section" " Networking: " 1000GB Public Bandwidth 1Gbps Public & Private Networks, None for Remainder Backend VLAN=1101,Frontend VLAN=4000 " Business Continuance: None Launch the SoftLayer VPN Client & connect from Navigate the SoftLayer Follow Links to CloudLayer -> Computing -> [CCI NAME] & Collect the eth0 Private VLAN IP Address & the administrator credentials. RDP from into the CCI/VM s Private VLAN IP (eth0) after the CCI has been provisioned. Download and install the traditional C based vsphere client from vsphereclient.vmware.com/vsphereclient/1/2/8/1/6/5/0/vmware-viclientall exe. It is possible to deploy the Virtual Center Server Appliance via Linux and the ovftool, this document will simplify the process with Windows usage of the traditional vsphere C Client to deploy vcenter. This utility VM will be leveraged to perform vcenter installation and other related image transfer services." 29 of 52

30 6. Configure Public&Private Ports (Port Control) Disable ESX Host Public Interfaces for Security Purposes Utilize the SoftLayer Follow Link to Public Network -> Port Control -> [esx Hostname] -> control" Select Disconnect for each ESX Hosts eth1 & eth3 pair for all hosts **This task can also be performed via the SoftLayer REST API reference/services/softlayer_hardware_server/shutdownpublicport." 7. Place SL Support Ticket to UnGroup the eth0 & eth2 pair for Management Cluster By Default, SoftLayer places NIC pair uplink ports on BCS & FCS switches into an LACP Pair. vsphere 5.1 & 5.5 only support NIC teaming with LACP on a vsphere Distributed Switch. In order to leverage IP Hash Teaming mechanisms on VMware@SoftLayer, a vsphere DVS & LACP must be utilized in conjunction with each other. This reference architecture will utilize vsphere Standard Switches for the Management Cluster ESX hosts Private Network uplinks to reduce scenarios where vcenter availability can affect Distributed Virtual Switch port availability. It will also leverage iscsi MPIO across both NICs on all ESX hosts, for these reasons, the port configuration on the BCS must be changed to ungrouped (unbunbled), and placed into the same VLAN (with same trunking, if applicable) for the Private Uplink eth0 & eth2 pair on the Management Cluster. This will enable proper load balancing for your vmnic teams. When not using LACP & DVS you MUST utilize Route based on the originating port ID for your teaming pairs to prevent MAC Flapping scenarios without LACP. The Capacity Cluster(s) will leverage full DVS architecture and can support LACP & Route based on IP hash teaming. Note: The Basic site architecture is utilizing Vyatta Appliances for North/South Public access network traffic and therefore eth1 & eth3 public links were disabled in step 6. If your architecture calls for Virtual Gateways (for example NSX Edge Gateways or Virtual Vyatta Gateways), the same guidance will apply for Public Ports you wish to enable. Public Ports can still be LACP managed if you place eth1 & eth3 on a DVS." 30 of 52

31 language=en_us&cmd=displaykc&externalid= " language=en_us&cmd=displaykc&externalid= " GUID-0D1EF5B B-B99D-5714B42CD7A9.html" Utilize the SoftLayer Follow Link to Support -> Tickets -> Add Ticket -> Create Standard Support Ticket" Subject = Private Networks Question Title = Disable LACP for VMware ESX Teaming Ticket Contents = Please change eth0 & eth2 LACP port channels to ungrouped (unbunbled), and placed into the same VLANs (with same trunking, if already applicable) w/out LACP for each host [list each Management Cluster host]. 31 of 52

32 8. Place SL Support Ticket for VLAN trunking Utilize the SoftLayer Follow Link to Support -> Tickets -> Add Ticket -> Create Standard Support Ticket" Subject = Private Networks Question Title = Trunk VLANs on eth0 & eth2 Ticket Contents = Please trunk VLANs 1001 & 1002 on eth0 & eth2 nic pair for for each host [list each host]. (Please note your VLAN ID s will likely be different) 9. Install ESX 5.5 on both Management ESX servers (KVM Method w BYOL) Refer to III"VMware vsphere ESX 4.x & 5.x Installation & Licensing -> Installation Methods in this document to deploy vsphere 5.5 Create default local Datastore on the Hosts Follow Links to HardWare -> Control Collect the eth0 Private IP for each server After Installation, assign the SoftLayer provisioned Private IP address & Appropriate Default Gateway from the VLAN1101 Primary Private Subnet to the vmk0 Management Interface on each ESX Server via the Remote Console (ikvm).(please note your VLAN ID s will likely be different) %2FGUID-7C9A1E23-7FCD CB1-C932F2423C63.html" 10. Configure vsphere Standard Switches/PortGroups for Management Cluster hosts At this point, either via the SoftLayer Management VPN ( or the Utility CCI, you should have access to Launch the C based traditional vsphere client and connect to the vmk0 Management IP Address you have assigned to the 2 Management Hosts during the previous step (Please note your VLAN ID s will likely be different). Configure/Create the following Basic Constructs ON Both ESX Hosts: vsphere Standard Switch vswitch0" vswitch Properties: Network Adapters = vmnic0 & vmnic2 vswitch Properties: NIC Teaming = Both adapters Active vswitch Properties: Load Balancing = Route based on the originating port ID Modify VM Port Group (0): Rename Existing VM Network Port Group: Name = VM-" " " Management Network" Modify VM Port Group (0): VLAN ID = None(0) Modify VMK Port(0)-vmk0:Rename Existing VMK Port:Name = VMK-Managment Modify VMK Port(0)-vmk0: VLAN ID = None(0) 32 of 52

33 Add VMK Port(1)-vmk1:Name = VMK-vMotion Network *** IP Address Does Not have to be from a SL Private Subnet as vmotion Traffic will not be routed, but all hosts in the cluster must have a vmotion IP Address in the same subnet. VMK Port(1)-vmk1:IP Address= #/24 VMK Port(1)-vmk1:VLAN ID = 1102 (Yours will likely be different) Add VMK Port(2):Name = VMK-FT Network *** IP Address Does Not have to be from a SL Private Subnet as FT Traffic will not be routed, but all hosts in the cluster must have a FT IP Address in the same subnet. VMK Port(2)-vmk2:IP Address= #/24 VMK Port(2)-vmk2:VLAN ID = 1102 (Yours will likely be different) Add VMK Port(3):Name = VMK-iSCSI Network PathA *** In Order to utilize MPIO, each vmk utilized for iscsi Traffic will be required to be linked to one physical vmnic, no teaming enabled. *** It is highly suggested to update the Notes section of each Portable IP Address with the name of the host & vmk port assigned. The Notes section can be located by navigating to the SoftLayer Management Private Network -> IP Manager -> [Subnet] VMK Port(3)-vmk3:IP Address=From Portable IP Subnet bound to VLAN1002 VMK Port(3)-vmk3:VLAN ID = 1102 (Yours will likely be different) VMK Port(3)-vmk3:NIC Teaming Load Balancing = enable Override vswitch & Use explicit Failover VMK Port(3)-vmk3:NIC Teaming Load Balancing = enable Override vswitch & vmnic0 = Active / vmnic2 = Unused Add VMK Port(4):Name = VMK-iSCSI Network PathB *** In Order to utilize MPIO, each vmk utilized for iscsi Traffic will be required to be linked to one physical vmnic, no teaming enabled. *** It is highly suggested to update the Notes section of each Portable IP Address with the name of the host & vmk port assigned. The Notes section can be located by navigating to the SoftLayer Management Private Network -> IP Manager -> [Subnet] VMK Port(4)-vmk4:IP Address=From Portable IP Subnet bound to VLAN1002 VMK Port(4)-vmk4:VLAN ID = 1102 (Yours will likely be different) VMK Port(4)-vmk4:NIC Teaming Load Balancing = enable Override vswitch & Use explicit Failover VMK Port(4)-vmk4:NIC Teaming Load Balancing = enable Override vswitch & vmnic2 = Active / vmnic0 = Unused CLI Reference: cli_manage_networks.11.1.html iscsi MPIO Reference: 33 of 52

34 11. Upload OS (Windows & Linux) ISO Images At this point, it is recommended to utilize a RDP session to the Utility CCI to download required images/ovfs for optimal performance. Download/Push the following ISO images to the Utility CCI: Windows Server 2008R2/Windows Server 2012 ISO Image (Your licensed Media) Centos Install Image ( Utilize the traditional vsphere Client or ssh to create a folder named ISO on the local datastore on one of the Management Hosts. Copy the ISO images to the ISO folder. 12. Upload vcenter Virtual Appliance (VCVA) At this point, it is recommended to utilize a RDP session to the Utility CCI to download required images/ovfs for optimal performance. Download/Push the following OVF to the Utility CCI: vcenter Server Appliance ( downloadgroup=vc550b&productid=351&rpid=4975) requires a valid VMware subscription (BYOL) 13. Deploy DNS (Windows Active Directory or BIND) Utilize the traditional vsphere client to create a VM on the Management ESX host where the ISO store is located. Connect the appropriate OS ISO (Windows/CentOS) to deploy a DNS server on the VM. It is beyond the scope of this document to cover AD/LDAP architecture, but either mechanism can be utilized dependent on the SL tenant s use case. Assign an IP address & Default Gateway from the Portable IP Subnet bound to VLAN 1101 *** It is highly suggested to update the Notes section of each Portable IP Address with the name of the VM assigned. The Notes section can be located by navigating to the SoftLayer Management Private Network -> IP Manager -> [Subnet] Connect the New VMs vmmic to the VM-Management Port Group Set DNS Forwarding to the service.softlayer.com local DNS hosts provided by SoftLayer: rs1.service.softlayer.com rs1.service.softlayer.com After DNS is setup, create a local DNS zone (example dal06.mycompany.local) && a reverse lookup zone for all Portable & Primary Subnets you have provisioned so far. Add an A HOST records for each Host s vmk0 Management IP Address Add an A HOST record from the Portable Subnet bound to VLAN 1101 for your vcenter Virtual Appliance That will be deployed in next step. Update the Notes Section of the Portable IP Subnet that you just assigned to vcenter. Windows AD/DSN HowTo: step-by-step-guide-for-setting-up-a-windows-server-2012-domain-controller.aspx CentOS BIND HowTo: 34 of 52

35 14. Deploy VCVA At this point, it is recommended to utilize a RDP session to Deploy the vcenter Virtual Appliance OVF downloaded in a previous step. Launch the traditional C based vsphere client and deploy the OVF: language=en_us&cmd=displaykc&externalid= " GUID-0B9988FF-5FB C58-EE3617B57E90.html" NOTE***At this point, there is no DHCP server available in the environment for the VCVA to assign an initial IP address. After initial power on you will see a console message similar to the image below: Simply login to the console (root/vmware) and exec /opt/vmware/share/vami/vami_config. This will allow the IP address assigned in DNS to be applied to the appliance. The VCVA will provide all required components of vcenter 5.5 (SSO, Inventory Service, vcenter Server, vsphere Webclient). During the configuration of the appliance the root password should be modified & NTP should be set to servertime.service.softlayer.com. 35 of 52

36 The IBM Cloud Innovation Lab chose the VCVA for this implementation due to its simplicity, form factor, and ease of installation. The VCVA is officially supported by VMware. However, it is possible and sometimes desirable to deploy vcenter on Windows. For example, in this architecture, the vcenter appliance will be provided resiliency by VMware HA services. For a more resilient architecture, it is possible to leverage vcenter Server Heartbeat ( which is only supported with a vcenter Windows Installation at the time of this writting. The VCVA can also be deployed via the OVFTOOL if a windows Instance to utilize the traditional C vsphere Client is not available. 15. Log In to VCVA & License Utilize the vsphere Web Client ( name/ip]/vsphere-client), log into the VCVA. Navigate to Administration -> Licenses Enter/Assign your VMware BYOL for vcenter/vsphere/vcloud Suite GUID-487AACBF-4E49-43E0-A852-FC23734C0774.html 16. Create vsphere DataCenter/Cluster Constructs Utilize the vsphere Web Client ( name/ip]/vsphere-client), log into the VCVA. Create The Following Objects: DataCenter: [Chosen SoftLayer DataCenter] Cluster(0):Management - 01 Cluster(1):Capacity - 01 Add both Mgmt Hosts to the Mgmt Cluster 17. Create DVSwitches & DV Port Groups Create a 5.5 dvswitch: Name = dvs-private w/ 2 Uplinks D21B3241-0AC9-437C-80B1-0C8043CC1D7D.html" 36 of 52

37 18. Create dvportgroups Utilize the vsphere Web Client ( name/ip]/vsphere-client), log into the VCVA. Create the Following Distributed Virtual Port Groups for the Capacity Cluster(s) on the dv- Private vswitch. dvpg-priv-vm Management Network VLAN = None Teaming & Failover= Route Based on IP Hash Uplink 1 & 2 both Active dvpg-priv-vmotion Network VLAN = 1102 (Yours will likely be different) Teaming & Failover= Route Based on IP Hash Uplink 1 & 2 both Active dvpg-priv-ft Network VLAN = 1102 (Yours will likely be different) Teaming & Failover= Route Based on IP Hash Uplink 1 & 2 both Active dvpg-priv-iscsi Network PathA VLAN = 1102 (Yours will likely be different) Teaming & Failover= Use Explicit Failover Order Uplink1 Active && Uplink2 Not Used dvpg-priv-iscsi Network PathB VLAN = 1102 (Yours will likely be different) Teaming & Failover= Use Explicit Failover Order Uplink2 Active && Uplink1 Not Used dvpg-priv-vm Access Network VLAN = 1103 (Yours will likely be different) Teaming & Failover= Route Based on IP Hash Uplink 1 & 2 both Active 37 of 52

38 19. Configure OSNexus/QuantaStor & Create iscsi Targets Utilize the SoftLayer Follow Link to Hardware -> Control -> [quantastore server] " Collect the admin/root password from the Software section Collect the IP address from eth0 in the Network Section Navigate to eth0 ip address] & login with credentials admin/[root password] Select Create Virtual Port in the QuantaStor Admin UI *** It is highly suggested to update the Notes section of the Portable IP Address with the name of the QuantaStor server and interface assigned. The Notes section can be located by navigating to the SoftLayer Management manage.coftlayer.com Private Network -> IP Manager -> [Subnet] IP Address = From Portable IP Subnet bound to VLAN1102 Subnet Mask = From Portable IP Subnet bound to VLAN1102 Gateway = From Portable IP Subnet bound to VLAN1102 Interface = bond0 38 of 52

39 Create a Storage Pool utilizing /dev/sdb. Navigate to Storage Pools & Select Create Name=StoragePool-01 Pool Type=Advanced (zfs) Raid Type=Raid 0 (Hardware is providing RAID) Storage=sdb IO Profile=Virtualization All Other Settings Default *If sdb is not available to add to a pool, It is due to a partition existing and being tagged as bootable, you will need to utilize gdsik on the Quantastor console to remove the partition and any /etc/fstab entries. Please file a support ticket with SoftLayer support." 39 of 52

40 Create 2 Volumes. These volumes will also create default iscsi LUN Targets in QuantaStor Navigate to Storage Volumes & Select Create" Volume 1 Name=Mgmt-Lun0 Storage Pool=StoragePool-01 Size=500GB All Other Settings Default Volume 2 Name=Capacity-Lun0 Storage Pool=StoragePool-01 Size=1.5TB %Reserved=50% All Other Settings Default 40 of 52

41 20. Mount iscsi Targets on the Management Cluster w/ iscsi MPIO Utilize the vsphere Web Client ( name/ip]/vsphere-client), log into the VCVA. Navigate to vcenter -> Hosts -> Manage -> Storage & collect the IQN for each iscsi adapter on the Management Hosts. If the iscsi adapter does not exist, click on the green + and add a Software iscsi adapter. Navigate to eth0 ip address] & login with credentials admin/[root password] Create Management Storage Group in QuantaStor so that the appropriate Luns may be masked. Navigate to Hosts & Select Create" Add Both Management Hosts" Select Add Host Name= [mgmnt cluster node name] Operating System Type=VMware iscsi Qualified Name (IQN):= [id from previous step] 41 of 52

42 Create Host Group, Navigate to Host Groups & Select Add. Name=ManagementCluster-01 Hosts=Select both Management Hosts Assign Volumes to Storage Groups, navigate to Host Groups & Select Assign. Host Group=ManagementCluster-01 Select Storage Volumes=Mgmtn-Lun0 of 42 52

43 Utilize the vsphere Web Client ( name/ip]/vsphere-client), log into the VCVA. Navigate to vcenter -> Hosts -> Manage -> Storage." Configure the vsphere iscsi Software Software Adapter: Network Port Binding = vmk3 & vmk4 (***also set Path Selection Policy to Round Robin) Targets Dynamic Discovery = [bond0:1 IP address assigned to QuantaStor in earlier step]. Create VMFS on the Lun & ensure it is mounted iscsi MPIO Reference: Configure the Following On the Management Cluster (NTP/DNS/Routing) Utilize the vsphere Web Client ( name/ip]/vsphere-client) Configure the following Options for each Management ESX Host DNS=[ip address of the DNS Server deployed in previous step] NTP=servertime.service.softlayer.com / Start With Server ***If You utilize SoftLayer Equalogic Based iscsi Storage, you may need to define static routes to ensure iscsi targets are mounted on iscsi vmks and not the Management default subnet route/interface. (example via DCUI: esxcfg-route -a [SL iscsi Subnet]/24 [Gateway on iscsi LAN]. DEB8086A-306B-4239-BF76-E FC.html 43 of 52

44 22. Modify SSH, Datastore Redundancy, Hardware Status Alerts in vcenter It may be helpful to disable certain vcenter Alerts at this point to reduce Alarms while you are setting up the environment: Disable SSH Warning Modify ESX Advanced Settings: UserVars.SuppressShellWarning change 0 to 1 Disable DataStore Redundancy Warning Add ESX Advanced Settings: das.ignoreinsufficienthbdatastore. = true Suppress Status of Other Host hardware Objects Warning (Prompted by Default SuperMicro BIOS Settings and SoftLayer Chassis Management tasks) 23. Deploy ESX 5.5 on 3 Capacity ESX servers (KVM Method w BYOL) Follow procedure outlined in Step 9: Install ESX 5.5 on both Management ESX servers (KVM Method w BYOL), To Deploy ESX 5.5 on the 3 x Management Hosts" 44 of 52

45 24. Configure the Following On the First Node in the Capacity Cluster (NTP/DNS/Routing) Follow procedure outlined in Step 21: Configure the Following On the Management Cluster (NTP/DNS/Routing). 25. Join the First Node in the Capacity Cluster to the dvswitch and create/migrate VMK IP Addresses Utilize the vsphere Web Client ( name/ip]/vsphere-client) Add the First Capacity Host to the Capacity-01 vsphere Cluster. Add the First Capacity Host ONLY (Management Host uplinks will remain on a Standard vswitch) to the dvs-private Distributed Virtual Switch: Assign vmmic0 to Uplink1 & vmnic2 to Uplink2 Migrate vmk0 to dvport Group= dvpg-priv-vm Management Network Create the following vmk Virtual Adapters for the First Host on dvs-private : *** It is highly suggested to update the Notes section of the Portable IP Addresses with the name of the Host and vmk interface assigned. The Notes section can be located by navigating to the SoftLayer Management Private Network -> IP Manager -> [Subnet] 26. Mount iscsi Storage on First Node in the Capacity Cluster. Follow procedure outlined in the Step 20: Mount iscsi Targets on the Management Cluster w/ MPIO. Basic Steps: vmk & Description dvport Group IP Address/Netmask Portable Subnet vmk1 (vmotion) dvpg-priv-vmotion Network Tenant Provided ex x/24 vmk2 (FT) dvpg-priv-ft Network Tenant Provided ex x/24 vmk3 (iscsi Path A) dvpg-priv-iscsi Network PathA from Subnet bound to 1102 (vmk3- vmk4 for iscsi) see step 1 for IP Subnets vmk4 (iscsi Path B) dvpg-priv-iscsi Network PathB from Subnet bound to 1102 (vmk3- vmk4 for iscsi) see step 1 for IP Subnets Add Host to QuantaStor Create Host Group CapacityCluster-01 Add Host to Host Group Capacity Cluster-01 Assign Capacity LUN to Host Group Capacity Cluster Create VMFS on the Lun & ensure it is mounted. 45 of 52

46 27. Capture Host Profile from First Node in the Capacity Cluster & Attach to the cluster. Utilize the vsphere Web Client ( name/ip]/vsphere-client) Add the remaining Capacity ESX Hosts to the Capacity Cluster Capture a vsphere Host Profile from the First Capacity Node Attach the Host Profile to the Capacity Cluster. Apply the Host Profile to Each remaining host, assign appropriate user variables (IP Addresses, etc ) 28. Configure DRS & HA on both the Management & Capacity Clusters Utilize the vsphere Web Client ( name/ip]/vsphere-client) Copy ISO images (Windows/Linux) from Utility VM ISO Folder to an ISO Folder on Mgmt iscsi Datastore & an ISO Folder on Capacity iscsi Datastore (Create both ISO folders on each Datastore) On Mgmt Cluster: svmotion VCVA & DNS server to Mgmt iscsi Datastore. On Capacity Cluster: Utilize ISO store to create a test VM, attach its vmic to dvpg-priv-vm Access Network, assign an IP address from Portable Subnet bound to VLAN *** It is highly suggested to update the Notes section of the Portable IP Addresses with the name of the Host and vmk interface assigned. The Notes section can be located by navigating to the SoftLayer Management Private Network -> IP Manager -> [Subnet] On Both Management & Capacity Clusters, Enable DRS & HA. Accept Default Settings. Test vmotion of AD VM in Management Cluster from Host1 to Host2. Test vmotion of Test VM in Capacity Cluster across all 3 Capacity Hosts. At this point a basic vsphere 5.5 Environment has been deployed. The Following Section will present details on securing the environment with a Vyatta Gateway. This basic architecture could also be protected with VMware SDN components such as VCNS & NSX, and will be covered in detail in VMware@SoftLayer CookBook: Advanced Connectivity. 46 of 52

47 29. Apply Vyatta Gateway Configuration These configuration steps will be accomplished by logging into the private IP of your Vyatta Gateway via the SoftLayer Management Utilize the Private Network -> Gateway Appliances -> View Gateway Appliances to collect the Management IP. (A) Configure bond interfaces to link to each VLAN & Subnet to be routed. The Vyatta ports will not be trunked & the IP addresses will not be accessible on the VLANs until a future step where the VLANS and their associated Subnets will be routed to the Vyatta. ssh into the Vyatta: configure set interfaces bonding bond0 vif 1101 address ##.###.###.###/## (Enter Default gateway of Primary Subnet Bound to VLAN 1101)" set interfaces bonding bond0 vif 1101 address ##.###.###.###/## (Enter Default gateway of Portable Subnet Bound to VLAN 1101)" set interfaces bonding bond0 vif 1102 address ##.###.###.###/## (Enter Default gateway of Portable Subnet Bound to VLAN 1102) set interfaces bonding bond0 vif 1103 address ##.###.###.###/## (Enter Default gateway of Portable Subnet Bound to VLAN 1103)" commit save 47 of 52

48 (B) Configure SNAT (if not still in configure mode) configure SNAT For CCI s (Utility) Bound to VLAN 1101 set nat source rule 10 set nat source rule 10 source address ##.###.###.###/## (Primary Subnet VLAN1101) set nat source rule 10 translation address ##.###.###.### (Vyatta bond1 IP) set nat source rule 10 outbound-interface bond1 SNAT For Management VMs Bound to VLAN 1101 set nat source rule 20 set nat source rule 20 source address ##.###.###.###/## (Portable Subnet VLAN1101) set nat source rule 20 translation address ##.###.###.### (Vyatta bond1 IP) set nat source rule 20 outbound-interface bond1 SNAT For Access VMs Bound to VLAN 1103 set nat source rule 30 set nat source rule 30 source address ##.###.###.###/## (Portable Subnet VLAN1103) set nat source rule 30 translation address ##.###.###.### (Vyatta bond1 IP) set nat source rule 30 outbound-interface bond1 commit save (C) Configure L2TP/IPSEC Remote Access VPN from MAC/Linux/Windows. (if not still in configure mode) configure set vpn ipsec ipsec-interfaces interface bond1 set vpn ipsec nat-traversal enable set vpn ipsec net-networks allowed-network /0 set vpn l2tp remote-access authentication local-users username (user u are creating) password (user password) set vpn l2tp remote-access authentication mode local set vpn l2tp remote-access client-ip-pool start (Start IP from non used CIDR) set vpn l2tp remote-access client-ip-pool stop (End IP from non used CIDR) set vpn l2tp remote-access dns-servers server-1 ###.###.###.### (Installed DNS server from Previous Step) set vpn l2tp remote-access dns-servers server-2 ###.###.###.### (Secondary DNS or SoftLayer DNS) set vpn l2tp remote-access outside-address ##.###.###.### (Vyatta bond1 IP) set vpn l2tp remote-access ipsec-settings authentication mode pre-shared-secret set vpn l2tp remote-access ipsec-settings authentication pre-shared-secret (Enter Shared Secret Key) commit save 48 of 52

49 Create Network Group Objects of IP Address ranges from Similar Security Zones. (D) Configure Firewall Groups (if not still in configure mode) configure set firewall group network-group SLSERVICES set firewall group network-group SLSERVICES network /19 set firewall group network-group SLSERVICES network /24 set firewall group network-group SLSERVICES network /24 set firewall group network-group SLSERVICES network /19 set firewall group network-group SLSERVICES network /19 set firewall group network-group SLSERVICES network /20 set firewall group network-group SLSERVICES network /20 set firewall group network-group SLSERVICES network /20 set firewall group network-group SLSERVICES network /20 set firewall group network-group SLSERVICES network /19 set firewall group network-group SLSERVICES network set firewall group network-group SLSERVICES network /24 set firewall group network-group SLSERVICES network /24 set firewall group network-group SLSERVICES network /24 set firewall group network-group SLSERVICES network /24 set firewall group network-group SLSERVICES network /24 set firewall group network-group SLSERVICES network /24 set firewall group network-group SLSERVICES network /24 set firewall group network-group SLSERVICES network /24 set firewall group network-group SLSERVICES network /24 set firewall group network-group SLSERVICES network /24 set firewall group network-group 1101PRIMARY network ###.###.###.### (Primary Subnet 1101) set firewall group network-group 1101VMMGMT network ###.###.###.### (Portable Subnet 1101) set firewall group network-group 1102PRIMARY network ###.###.###.### (Primary Subnet 1102) set firewall group network-group 1102VMKISCSI network ###.###.###.### (Portable Subnet 1102) set firewall group network-group 1103VMACCESS network ###.###.###.### (Portable Subnet 1101) commit save 49 of 52

50 (E) Configure Firewall Name Rules Define Firewall Rules for each direction of Traffic. (if not still in configure mode) configure set firewall name INSIDE2OUTSIDE set firewall name INSIDE2OUTSIDE default-action drop set firewall name INSIDE2OUTSIDE rule 10 action accept set firewall name INSIDE2OUTSIDE rule 10 protocol all set firewall name INSIDE2OUTSIDE rule 10 source group network-group 1101VMMGMT set firewall name INSIDE2OUTSIDE rule 20 action accept set firewall name INSIDE2OUTSIDE rule 20 protocol all set firewall name INSIDE2OUTSIDE rule 20 source group network-group 1103VMACCESS set firewall name OUTSIDE2INSIDE set firewall name OUTSIDE2INSIDE default-action drop set firewall name OUTSIDE2INSIDE rule 10 action accept set firewall name OUTSIDE2INSIDE rule 10 protocol udp set firewall name OUTSIDE2INSIDE rule 20 action accept set firewall name OUTSIDE2INSIDE rule 20 protocol udp set firewall name OUTSIDE2INSIDE rule 20 destination port 4500 set firewall name OUTSIDE2INSIDE rule 30 action accept set firewall name OUTSIDE2INSIDE rule 30 protocol udp set firewall name OUTSIDE2INSIDE rule 30 destination port 500 set firewall name OUTSIDE2INSIDE rule 40 action accept set firewall name OUTSIDE2INSIDE rule 40 ipsec match-ipsec set firewall name OUTSIDE2INSIDE rule 50 action accept set firewall name OUTSIDE2INSIDE rule 50 protocol gre set firewall name OUTSIDE2INSIDE rule 60 action accept set firewall name OUTSIDE2INSIDE rule 60 protocol tcp set firewall name OUTSIDE2INSIDE rule 60 destination port 1723 set firewall name OUTSIDE2INSIDE rule 70 action accept set firewall name OUTSIDE2INSIDE rule 70 protocol tcp set firewall name OUTSIDE2INSIDE rule 70 destination port 80 set firewall name OUTSIDE2INSIDE rule 80 action accept set firewall name OUTSIDE2INSIDE rule 80 protocol tcp set firewall name OUTSIDE2INSIDE rule 80 destination port 443 set firewall name OUTSIDE2INSIDE rule 90 action accept set firewall name OUTSIDE2INSIDE rule 90 state established enable set firewall name SLSERVICE2INSIDE set firewall name SLSERVICE2INSIDE default-action drop set firewall name SLSERVICE2INSIDE rule 10 action accept set firewall name SLSERVICE2INSIDE rule 10 protocol all set firewall name SLSERVICE2INSIDE rule 10 source group network-group SLSERVICES set firewall name INSIDE2SLSERVICE set firewall name INSIDE2SLSERVICE default-action drop set firewall name INSIDE2SLSERVICE rule 10 action accept set firewall name INSIDE2SLSERVICE rule 10 protocol all set firewall name INSIDE2SLSERVICE rule 10 destination group network-group SLSERVICES 50 of 52

51 set firewall name L2TP2MGMT set firewall name L2TP2MGMT default-action drop set firewall name L2TP2MGMT rule 10 action accept set firewall name L2TP2MGMT rule 10 protocol all set firewall name L2TP2MGMT rule 10 source group 1101VMMGMT set firewall name MGMT2L2TP set firewall name MGMT2L2TP default-action drop set firewall name MGMT2L2TP rule 10 action accept set firewall name MGMT2L2TP rule 10 protocol all set firewall name MGMT2L2TP rule 10 destination group 1101VMMGMT set firewall name VMACCESS2MGMT set firewall name VMACCESS2MGMT default-action drop set firewall name VMACCESS2MGMT rule 10 action drop set firewall name VMACCESS2MGMT rule 10 protocol all set firewall name VMACCESS2MGMT rule 10 source group 1103VMACCESS commit save (F) Configure Zone bindings (if not still in configure mode) configure set zone-policy zone OUTSIDE description Internet Zone set zone-policy zone OUTSIDE default-action drop set zone-policy zone OUTSIDE interface bond1 set zone-policy zone SLSERVICE description SoftLayer Services set zone-policy zone SLSERVICE default-action drop set zone-policy zone SLSERVICE interface bond0 set zone-policy zone MGMT description Management VMs & ESX Host Access set zone-policy zone MGMT default-action drop set zone-policy zone MGMT interface bond set zone-policy zone VMACCESS description VM Access set zone-policy zone VMACCESS default-action drop set zone-policy zone VMACCESS interface bond set zone-policy zone L2TP description Remote VPN Access set zone-policy zone L2TP default-action drop set zone-policy zone L2TP interface l2tp+ (Error Prompt can be ignored) commit save (G) Configure Zone-Policy (if not still in configure mode) configure set zone-policy zone OUTSIDE from MGMT firewall name INSIDE2OUTSIDE set zone-policy zone OUTSIDE from VMACCESS firewall name INSIDE2OUTSIDE set zone-policy zone VMACCESS from OUTSIDE firewall name OUTSIDE2INSIDE set zone-policy zone MGMT from OUTSIDE firewall name OUTSIDE2INSIDE set zone-policy zone SLSERVICE from MGMT firewall name INSIDE2SLSERVICE set zone-policy zone MGMT from SLSERVICE firewall name SLSERVICE2INSIDE set zone-policy zone MGMT from L2TP firewall name L2TP2MGMT set zone-policy zone L2TP from MGMT firewall name MGMT2L2TP 51 of 52

52 (H) Route Private & Public Subnets After the Vyatta has been configured, The VLANs to be protected will have to be routed to the Vyatta gateway VLANS. This process will disable the existing SoftLayer default routing, the existing default gateways will be removed from the VLANS (this is why the same default gateways IP Addresses were assigned to the Vyatta bond0.#### interfaces). Ensure the Vyatta configuration is correct as connectivity may be lost to the subnets located in each VLAN. if the configuration is not correct. Utilize the Private Network -> Gateway Appliances - > [gateway appliance]" Navigate to the Associated VLANs section & Select Action = Route. This task will trunk the VLANs and hand off routing of the associated subnets to the Vyatta. If All steps have been completed properly, a functional Basic vsphere implementation should now exist within your SoftLayer DataCenter. 52 of 52