Intel Cloud Builders Guide to Cloud Design and Deployment on Intel Platforms

Transcription

1 Intel Cloud Builders Guide Intel Xeon Processor-based Servers Power Capping Impact to the Network: 10GbE Intel Ethernet Converged Network Adapter X540 with Intel Node Manager and VMware* vsphere 5.0* Intel Cloud Builders Guide to Cloud Design and Deployment on Intel Platforms Power Capping Impact to the Network: 10GbE Intel Ethernet Converged Network Adapter X540 with Intel Node Manager and VMware* vsphere 5.0* Intel Xeon E5 Processor E5 Family Intel Ethernet Adapters Audience and Purpose This reference architecture outlines a usage of energy management and network technologies as part of planning, provisioning, and optimizing strategies in enterprise and cloud data centers to reduce energy cost and address constrained power situations while realizing 10 Gigabit Ethernet (10GbE) throughput in their environments. This paper will give IT professionals architectural insights with methodical instructions to demonstrate Intel Manageability products, run end-to-end on VMware* vsphere* 5.0 and 10GbE on Intel Xeon processor-based servers. It is intended for data center administrators and enterprise IT professionals who seek energy management solutions to achieve better energy efficiency and power capacity utilization within existing or new data centers. This study also shows the relationship between processor, network throughput, and power management. Our conclusions can inform the IT professional in how understanding the impact of power management on network throughput can bring agility and reduced costs to their environments. January 2013

2 Table of Contents Executive Summary...3 Introduction...3 Test Configuration and Environment Overview...4 Detailed Hardware Configuration...5 Data Center Power Management...6 Details Steps for Baseboard Management Controller (BMC)...7 Details Steps for Intel Data Center Manager (Intel DCM)...10 Establishing Thresholds...15 Use Case Overview...16 Use Case One...16 Use Case Two...17 Use Case Three...17 Usage Model Test Results and Analysis...18 Conclusion...21 References

3 Executive Summary Cloud data centers require focused planning in regards to density and environmentals. This paper is intended to define the relationship of server power to network throughput in a virtualized environment. To maintain a highly efficient and optimized environment, we ll limit power to the test environment to show the impact to network traffic to our System under Test (SUT) in a virtualized environment. Our goal is to create optimized and energy-efficient usage model that can reduce power costs in the data center. Our requirements include the capability to monitor and regulate power in real-time at server, rack, zone, or data center levels. This means the ability to monitor and manage aggregated power consumption within a rack, zone, or data center based on available power and cooling resources. We ll use standard network load generation tools to drive the limits of the Intel Ethernet Converged Network Adapter X540-T2. Our workload profiles are built for maximum throughput to determine how much network capping can be applied before the performance degenerates. The approach is to match actual performance against the lowest power level setting. This showcases the ability to enforce power SLAs across different populations of users with differing workloads. Multiplying this effect in data centers with hundreds or thousands of servers can significantly reduce costs of power and cooling. To support this evolution, Intel works with end users to create an open data center roadmap of usage models that address key IT pain points for more secure, efficient, and simple cloud architectures built on a foundation of transparency. This paper describes the configuration details for setting up the virtualized scenarios, along with the functionality and performance observed in our lab testing. Commonly used tools iperf and NTttcp are used to achieve near line rate performance in most cases. This work is meant to bring new insights on the use of native networking initiators. Introduction At its core, a data center is a centralized repository for storage, management, and dissemination of data. IT and enterprise managers are always challenged to finds efficiencies and cost reduction in regards to their datacenters all while maintaining data integrity. Virtualization I/O performance is one of the key considerations in IT shops. Virtualization increases efficiencies in the datacenter through it impacts on environmental systems like power, cooling, and spacing. As IT departments look to reduce costs and improve server efficiency, they are turning increasingly towards server virtualization. If you look at the trend of virtualization deployments in the last decade or so, initially it started with consolidation of software development and testing environments and moved on to consolidation of production applications. High-speed networking has quickly become a hot topic amongst IT professionals. As network transport speeds increase, LAN and storage consolidation becomes achievable and desirable. This is largely due to consolidation of system resources and reduction of network complexities. Regardless of whether this is a traditional or cloud data center, creation of a seamless data transport is of utmost importance to any data center administrator. Management tools for the data center have become very sophisticated in the granularity to which they control their environments. In addition, more systems can be managed by fewer administrators. Intel Node Manager is an excellent example. Intel Node Manager was implemented on Intel server chipsets starting with Intel Xeon processor 5500 series platforms. After significant improvements, Intel Node Manager 2.0 was introduced in 2012 on server platforms supporting the Intel Xeon processor E5 family (presented in this document). The success of this experiment was yielded from key Intel platform hardware capabilities such as: Real-time Server Energy Usage Monitoring, Reporting, and Analysis to get continuous and actual energy usage visibility via agentless monitoring of the servers along with other devices and systems in the enterprise network, data center, and facilities. Power Guard Rail and Optimization of Rack Density by imposing power guard to prevent server power consumption from straying beyond a preset limit. Disaster Recovery/Business Continuity by applying significantly lower power caps to reduce power consumption and heat generation when unforeseen circumstances like power outage and cooling system failure occurs. Intel QuickData Technology enables data copy by the chipset instead of the CPU, to move data more efficiently through the server and provide fast, scalable, and reliable throughput. Intel Integrated I/O and Intel Data Direct I/O Technology (Intel DDIO) increases bandwidth, lowers latency, and reduces power consumption through direct communication between the NIC and the processor cache. Extended Message Signaled Interrupts (MSI-X) distributes I/O interrupts to multiple CPUs and cores, for higher efficiency, better CPU utilization, and higher application performance. 3

4 Receive Side Coalescing (RSC) aggregates packets from the same TCP/ IP flow into one larger packet, reducing per-packet processing costs for faster TCP/IP processing. Low Latency Interrupts tune interrupt interval times depending on the latency sensitivity of the data, using criteria such as port number or packet size, for higher processing efficiency. The purpose of this reference architecture is to demonstrate that 10GbE line speed can be achieved while regulating power to the system or systems, to show how to setup and configure different parameters, and to serve as a reference for end users in deploying these technologies in their virtualized environment. Test Configuration and Environment Overview Our physical environment at its core is simple. We use two Intel Xeon processor E series-based servers running VMware* ESXi* 5.0 u1.0 with five virtual machines (VMs) on each server. Each VM is run from an iscsi shared volume on an EMC* VNX 5500 storage array. All systems are attached by a Cisco* Nexus 5548 switch and Cisco Nexus* 2232TM Fabric Extension Module (FEX) for the performance portion of the testing and an Extreme X460 for remote access and hosts management. 10GbE server connections were made with the Intel Ethernet Converged Network Adapter X540-T2 and 1GbE connection were made with the onboard Intel Ethernet Server Adapter I350-F2. See Table 1 for details of the systems in this setup. Figure 1: Test Environment Overview 4

5 Detailed Hardware Configuration Intel Xeon E5 Processor Family Server Under Test (SUT) Intel Xeon processor 2.70 GHz (2 sockets, 8 cores/socket), SMT enabled, 64Gb RAM, 120GB SSD VMware* ESXi* 5.0 build VM 20GB VMDK, 4 vcpu, 4GB RAM, W2K8 R2, 5GB Secondary Storage VMware vsphere* Server 5.0 build VMware vsphere Client 5.0 build Data Center Management Intel Data Center Manager Network Configuration Cisco Nexus 5548 Chassis 32x10GE/Supervisor, Image version 5.1(3)N1(1) Cisco Nexus 2232TM FEX 32x10G Base-T Intel Ethernet Converged Network Adapter X540-T2, version: NAPI Intel Ethernet Server Adapter I350-F2, version: VMware ESXi Clustered vsphere Distributed Switch (VDS) VMware ESXi Native iscsi Initiator Storage Configuration EMC VNX* 5500 Storage System (Unified); Version: TB Capacity, 12GB Memory EMC Control Station Linux release 3.0 ( NAS ) Cluster and VDS Configuration Application Configuration Data Collection Intel R2300GZ Server (S2600GZ Server Board) Cluster PCN_VDS_X540 IOmeter v # of Managers = 1 # of Workers = 8 # of Outstanding I/O 8 I/O Size = 1MB NTttcp v iperf v CPU utilization, storage I/O, and network throughput captured from esxtop Table 1: Setup and Configuration 5

6 Data Center Power Management The Baseboard Management Controller (BMC) and detailed steps for Intel Data Center Manager (Intel DCM) are the focus of this configuration. Servers, virtual machines, switches, storage, and test programs have to be configured but are outside the scope of this paper. The BMC architecture is shown in Figure 2 below is configured as described below. 1. In the BIOS, configure BMC network settings with static or DHCP option as desired, and provide the BMC hostname. We used DHCP. Note the BMC hostname or IP address. 2. Note the user name and password of the BMC user with administrator privileges. Either use root user ensuring administrator privileges are granted, or add another user. 3. Install the operating system, applications, and workloads as desired. 4. Visual Data Center connects to the servers out of band with BMC hostname and log in credentials. There is no agent installed on the server. Figure 2 6

7 Detailed Steps for Baseboard Management Controller (BMC) 1. Press F2 to get into the BIOS on server startup navigate to the Server Management Tab and select BMC LAN Configuration. 2. Select the IP source to configure a Dynamic or Static for the management IP address. 7

8 3. Select the BMC DHCP Host Name to configure a unique hostname for the management IP. 4. Select the User ID to ensure that a login name exists. 8

9 5. Select the User Password to set a password for management access. 6. Configuration for Intel DCM power manageability is complete. 7. (Optional Step) If there is a Remote Management Module (RMM) installed on this system, the user can open an Internet browser and type in the BMC IP address noted above; in this example This will open a login window to the server management interface. Login with the default username/password setup in your documentation. 9

10 8. (Optional Step) A successful login will take the user to the WebUI. Detailed Steps for Intel Data Center Manager (Intel DCM) 1. Login to the Intel DCM via HTTP or directly through the application executable on the server desktop. RT+Click in open below Desktop Hierarchy to add the servers to be managed. 10

11 2. Add the server management details that were setup in the BMC: a. Entity Name b. Type: Server c. Connection Name d. BMC IP Address e. BMC Username/Password f. Derated Power or the graph threshold 11

12 3. Intel DCM now is configured to take basic trending. 4. For this use case, we will need to group the servers. RT+Click in open below Desktop Hierarchy to add a group. 12

13 5. Name the group and give it a type. Types can be room, row, rack, or data center. Click 'save' to continue. 6. Simply drag the server or servers into the group. 13

14 7. RT+Click the new group to add a policy. At the Policy Lists window, select New to add a new policy. 8. At the Policy Details window, set the Policy Type, Policy Entity, Policy Threshold, and Policy Reserve. Click save to return to the Policies List window. Enable the policy and click close to return to Intel DCM. 14

15 9. The power cap and average server groups power are now clearly seen. Note that these values do not reflect any load. Establishing Thresholds Thresholds are guidelines that administrators can use to establish common boundaries for compute monitoring. The reader should understand that the power and temperature thresholds used in this experiment were created to push the limits of network and storage throughputs and should not be used as a guideline. Users recreating these experiments should develop their own methods to determine the power and temperature thresholds as each data center is unique and generalizations should not be used. Tests Subtests: Network Subtests: Storage + Network 1. Baseline No Power Cap 1x VM 10Gb Receive (Rx) 5x VM 10Gb Storage Read 2. Power Restriction: 425W 1x VM 10Gb Transmit (Tx) 5x VM 10Gb Storage Write 3. Power Restriction: 350W 1x VM 10Gb Bidirectional (Bx) 1x VM 10Gb Rx + Read 4. Power Restriction: 300W 5x VM 10Gb Receive (Rx) 1x VM 10Gb Tx + Write 5. Power Restriction: 275W 5x VM 10Gb Transmit (Tx) 5x VM 10Gb Bidirectional (Bx) 5x VM 10Gb Rx + Read 5x VM 10Gb Tx + Write 15

16 Use Case Overview IT organizations (including Intel IT) face significant data center power and cooling challenges. So, companies seek alternative approaches that focus on more efficient use of existing data center power. Power optimization of the workloads is one such approach to achieve power efficiency. Power optimization an understanding of the workload profile and a performance loss target, if any, it is not to be exceeded. Developers perform a series of experiments to characterize how much capping can be applied before the performance target is hit. Afterwards, during normal operations, the applications engineer sets power capping targets based on the prior measurements. The system is now said to be optimized, because the impact of the application of these caps is now known. Use Case One: Monitoring Power, Temperature, and Events of a Device or Group The most basic use case of monitoring gives a data baseline to evaluate the systems in the Intel DCM group. It gives enough data to make conscious decision for power limiting. This proves very useful as data center administrators are expected to have information on rack density, thermal footprint, and overall power consumption on a measured basis across their data centers. These data points can be aggregated to make critical decisions on server placement, manage ways to optimize cooling, and also maximize the usage of power circuits throughout the room. The figure below shows the server group running at idle for the majority of the time and some moderate workload, all while sustaining a 425W static power cap. Figure 3: Server Group at Idle with 425W Static Power Cap 16

17 Use Case Two: Creating Power Optimized Policies to Understand Power Efficiencies At the heart of this experiment, power policies are created in Intel Node Manager that restrict or power cap the server group in a linear fashion to understand the impact to network throughput. This is done to identify power optimization points and power efficiencies. While these policies can be set static or dynamic, all policies were static. The initial baseline tests were done without power restrictions with the following tests restricting power incrementally. In all, five power cap tests were run with up to twelve subtests per power cap test. The example below shows a 300W power cap test with all 12 subtests. Figure 4: 300W Power Cap with 12 Subtests Use Case Three: Varied vcpu and Memory Configurations to Understand Power Efficiencies Each test group was run with VMs in two differing configuration. The first set of tests was run on one and five VM runs, each VM configured with four vcpu and 4GB memory. The second set of tests was run on one and five VM runs, each VM configured with six vcpu and 12GB memory. These test groupings were derived to see the impact of power restrictions to VMs with varying power needs. Tests Subtests: Network Subtests: Storage + Network 6. Baseline No Power Cap 1x VM 10Gb Receive (Rx) 5x VM 10Gb Storage Read 7. Power Restriction: 425W 1x VM 10Gb Transmit (Tx) 5x VM 10Gb Storage Write 8. Power Restriction: 350W 9. Power Restriction: 300W 10. Power Restriction: 275W 1x VM 10Gb Bidirectional (Bx) 5x VM 10Gb Receive (Rx) 5x VM 10Gb Transmit (Tx) 5x VM 10Gb Bidirectional (Bx) 1x VM 10Gb Rx + Read 1x VM 10Gb Tx + Write 5x VM 10Gb Rx + Read 5x VM 10Gb Tx + Write 17

18 Usage Model Test Results and Analysis As mention above, this reference architecture has three primary use cases; baseline power, optimized policies, and varied processor and memory. At completion there were 120 tests runs. Upon analysis of each test group, it was decided to focus on the five VM receive and transmit tests in both four vcpu and six vcpu groups. This first set of graphs below show the actual ESXTOP VMNIC network throughput and the host CPU trending for each power cap test. Figure 5: Graphed Results of Five VM Tests; Four vcpu/4gb RAM and Six vcpu/12gb RAM 18

19 While this paper focuses on the series of five VM tests, a baseline of one VM test was needed. This next set of graphs below show the average network throughput and average CPU utilization for both one VM and five VM tests in both four vcpu and six vcpu configurations. Figure 6: 1x VM Average Network Throughput and CPU Utilization; Four vcpu/4gb RAM and Six vcpu/12gb RAM 19

20 Figure 7: 5x VM Average Network Throughput and CPU Utilization; Four vcpu/4gb RAM and Six vcpu/12gb RAM The third set of graphs below show the relative variance from baseline for the five VM test groups. Each power cap group s (425W, 350W, 300W, and 275W) average network throughput and CPU utilization was divided by the baseline average. Just to note, the axis value of 100 is actually is really the value of the baseline i.e. the 425W network throughput value for Rx 4x4 shows the difference between 9.2Gbs and 9.1Gbs, the average value of network throughput of the 425W test. The same is done for CPU utilization. Figure 8: Efficiency Variances to Baseline 20

21 Analysis Starting with the first set of data charts, we see all five VM test achieved linespeed at a very steady state in both network throughput and CPU utilization. We see from comparison of the second set of charts that we achieve a better efficiency in scale between the one VM baseline test and the five VM tests overall. We note that for this two server group, 300W is the teetering point. Our workload began to take longer to complete with a power cap for less or equal to 300W with the 275W test time extending up to 15 percent. Additionally, at a power cap of less than 300W, we begin to see network throughput thrash and loss. We saw no significant difference in the impact to transmit or receive network throughput at power cap settings greater than 300W noting a variance of approximately 2 percent loss of network throughput in the greater than or equal to 300W tests. We didn t see value in getting more granular within 25W. As expected, CPU utilization was more noticeable throughout the power cap tests due to the direct relationship between electric power and processor power. We also noted that VM network throughput on the VMs with higher vcpu and memory (6 vcpu and 12 GB RAM) assignments saw a larger impact by the power cap. Yet with the linear stepping down of power states for each power cap test, workloads at greater to or equal to 300W were minimally impacted. Additionally we saw the CPU states become less steady at less than 300W power cap tests. percent network throughput loss and a 35 percent reduction to overall average CPU utilization. Conclusions With companies consistently focused on lowering total cost of ownership (TCO) while still meeting customer demands for increased capability, the benefit of tuning your environment is crucial. The test results noted in this paper are just one way to optimize your datacenters to meet operational efficiencies. Intel components are becoming faster, power-efficient, and manageable offering administrators more control of their enterprise environments while meeting or exceeding computing requirements. We believe our test environment is scalable, manageable, and efficient. Setting power limits removes the highcost power peaks in data center racks and the results of these tests were able to reduce power significantly while keeping network throughput line-rates. More Intel/VMware reference architectures are available at the Intel Cloud Builders Web site at cloudbuilders. Our baseline for the two server group was 520W. In power cap steps for 425W, 350W, and 300W, all network throughput test reduction was within 2 percent max total. This translates to a reduction of overall power to the group by 43 percent yielding a less than a 2 21

22 References Data Center Energy Efficiency with Intel Power Management Technologies - IT@Intel Brief, February Gb Classic Fibre Channel vs. 10GbE Unified Networking in a Virtualized Environment Intel Cloud Builders Reference Architecture 2011: Data Center Infrastructure and Energy Management with 3-D Visualization with Visual Data Center* Intel Cloud Builders Reference Architecture Data Center Energy Management with Dell, Intel, and ZZNode* Intel Cloud Builders Reference Architecture intelcloudbuilders.com/docs/intel_cloud_builders_dell_zznode.pdf Data Center Energy Management Dell, Intel, and JouleX Intel Cloud Builders Reference Architecture intelcloudbuilders.com/docs/intel_cloud_builders_dell_joulex_data_center_energy_march_2012.pdf Microsoft System Center Virtual Machine Manager Self-Service Portal* 2.0 with Policy-Based Power Management Intel Cloud Builders Reference Architecture July2011.pdf What s new in VMware vsphere Performance: Performance-Technical-Whitepaper.pdf What s new in VMware vsphere 5.0 Networking: Networking-Technical-Whitepaper.pdf Intel Cloud Builders Reference Architecture Library: Iometer: Website: Intel Reference Site: Unified Networking Reference Site: intel-offers-unified-networking-technology/ Intel Virtualization Technology for Connectivity Improving Network Performance in Multi-Core Systems Intel Reference Site: Intel Product Site: DCB: Wikipedia Reference Site: Wikipedia Reference Site: 22

23 Disclaimers Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. See products/processor_number for details. INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked reserved or undefined. Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling , or by visiting Intel s Web site at Copyright 2013 Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon, Xeon inside, and Intel Intelligent Power Node Manager are trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others.