HUAWEI OceanStor 9000 Load Balancing Technical Issue 01 Date 2014-06-20 HUAWEI TECHNOLOGIES CO., LTD.
Copyright Huawei Technologies Co., Ltd. 2014. All rights reserved. No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd. Trademarks and Permissions and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders. Notice The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied. The information in this document is subject to change without notice. Every effort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied. Huawei Technologies Co., Ltd. Address: Website: Huawei Industrial Base Bantian, Longgang Shenzhen 518129 People's Republic of China http://enterprise.huawei.com Issue 01 (2014-06-20) Huawei Confidential i
HUAWEI HUAWEI OceanStor 9000 Load Balancing Technical Contents 1 Overview... 3 2 Technical Principles... 4 3 Highlights... 6 3.1 Load Balancing Based on Reliable Domain Name Resolution... 6 3.2 Intelligent IP Address Management... 7 3.3 Diverse Load Balancing Policies... 9 3.4 Zone-based Node Management... 12 4 Customer Benefits... 13 5 Acronyms and Abbreviations... 14 Issue 01 (2014-06-20) Huawei Confidential ii
1 Overview Load balancing evenly allocates loads (jobs) to multiple units, such as web servers, File Transfer Protocol (FTP) servers, critical application servers, and other critical task servers, to enable them to complete the work jointly. Load balancing is based on the existing network structures. It is a cost-effective and efficient way to expand the bandwidth of network devices and servers, increase throughput, improve the data processing capability, and enhance network flexibility and availability. Common load balancing techniques are as follows: DNS-based load balancing: The earliest load balancing is implemented based on a domain name system (DNS). In the DNS, the same name is allocated to multiple addresses. A client that queries the name obtains one of the addresses. In this way, different users access different servers, achieving load balancing. Proxy server based load balancing: A proxy server can be used to forward requests to an internal server, accelerating access to static web pages. A proxy server can also be used to forward requests to multiple serves evenly for load balancing. ATG-based load balancing: An address translation gateway (ATG) that supports load balancing is able to map one external IP address to multiple internal IP addresses. For each TCP connection request, the ATG dynamically uses one of the internal addresses to achieve load balancing. Protocol-based load balancing: Some protocols provide functions related to load balancing, such as the redirection capability of HTTP that runs at the highest layer of a TCP connection. For load balancing based on unified IP addresses, all service packets must be forwarded through load balancing service nodes. For this reason, load balancing service nodes tend to become performance bottlenecks. To mitigate the impact of the load balancing service on system performance and to provide domain name access externally, the OceanStor 9000 InfoEqualizer, a client connection load balancing feature, employs a DNS-based load balancing technique. During domain name resolution, the OceanStor 9000 load balancing service allocates one clustered node to a client based on a specific load algorithm. The subsequent service data exchange is completed between the client and the node allocated to the client. Issue 01 (2014-06-20) Huawei Confidential 3
2 Technical Principles The following figure shows a schematic diagram of the load balancing service. Figure 2-1 Schematic diagram of the load balancing service User Domain name request User Domain name request Domain name request Enterprise domain name service Node 1 Node 2 (primary node) Node 3 Reporting load information Reporting load information The OceanStor 9000 load balancing service is designed as a clustered system. In the initial phase of cluster formation, OceanStor 9000 nodes negotiate based on the Paxos algorithm to determine a node as the primary node. In the cluster, there is always only one primary node at any time. Each node periodically collects its load information, including the number of CPU cores, CPU dominant frequency, memory size, network adapter information, current CPU usage, current memory usage, current network throughput, and number of current NAS client connections. Each node sends the collected load information to the primary node in a unified manner. The primary node uses the collected load information as the basis of load balancing. The OceanStor 9000 provides unified domain name access externally. The domain name query request service is integrated with the load balancing service. When a client initiates a domain name query request, the load balancing service calculates the load based on the configured load balancing policy, and then returns the IP address of an appropriate node to the client for accessing the OceanStor 9000. If a client does not have a DNS server, the DNS IP address of the local host can be set as the DNS IP address of the OceanStor 9000 for accessing the DNS service of the OceanStor 9000. If a client has a DNS server, the domain name forwarding function can be configured Issue 01 (2014-06-20) Huawei Confidential 4
on the server to forward the domain name request for accessing the OceanStor 9000 to the DNS service of the OceanStor 9000. Issue 01 (2014-06-20) Huawei Confidential 5
3 Highlights 3.1 Load Balancing Based on Reliable Domain Name Resolution The following figure shows a schematic diagram of the access to the OceanStor 9000. Figure 3-1 Schematic diagram of the access to the OceanStor 9000 User 1. Domain name query request 2. Response to the domain name query request 3. Service data access Storage system Node 1 Node 2 Node 3 The OceanStor 9000 load balancing service is based on domain name requests. The service works only when domain names are requested and does not participate in actual data flow services. Therefore, the load balancing service will not be the system performance bottleneck. Different from most DNS-based load balancing technologies, the OceanStor 9000 load balancing service has the DNS query response function integrated. Therefore, an extra DNS service is not needed. In addition, the OceanStor 9000 load balancing service has the following DNS optimizations: Most DNS-based load balancing technologies require that an IP address be configured for a domain name in advance. If a node is added or abnormal, the DNS configuration must be updated and the DNS service must be restarted. The DNS integrated with the OceanStor 9000 load balancing service, however, does not rely on any configuration. The OceanStor 9000 load balancing service is invoked to calculate the mapping between a domain name and an IP address. The OceanStor 9000 load balancing service periodically queries information about clustered nodes. If a clustered node is abnormal, the capacity is expanded, or the load varies, information is updated immediately without manual intervention or configuration. Issue 01 (2014-06-20) Huawei Confidential 6
Most DNS-based load balancing technologies have a client cache problem, causing load imbalance to some extent. To resolve this issue, the OceanStor 9000 load balancing service dynamically adjusts the time to live (TTL) for DNS query responses, minimizing the client cache time. Most DNS-based load balancing technologies are vulnerable to single points of failure. The OceanStor 9000 load balancing service has the DNS query function integrated and works as a cluster. If the current load balancing service is abnormal, another node in the cluster takes over its service. Also, the DNS IP address provided for user configuration is switched to the new node. Load balancing service nodes are reliable and transparent to users, minimizing the impact on users. 3.2 Intelligent IP Address Management The OceanStor 9000 manages access IP addresses externally provided by clustered nodes in a unified manner. The OceanStor 9000 automatically allocates an IP address to a newly added node and supports failover and failback of node IP addresses. A user only needs to configure an IP address pool for the OceanStor 9000, instead of allocating an IP address to each node one by one. This management method simplifies IP address management and facilitates cluster expansion. Each OceanStor 9000 node has a static IP address and a dynamic IP address. Both of them can be used to access the OceanStor 9000 service. After a failed node recovers, its static IP address remains the same. However, its original dynamic IP address is lost, and a new dynamic IP address will be allocated to the node. During environment deployment, a deployment tool is used to configure static IP addresses in a unified manner. Dynamic IP addresses are allocated by the load balancing service in a unified manner based on an IP address pool. The following figure shows a schematic diagram of how IP addresses are allocated to nodes. Figure 3-2 Schematic diagram of how IP addresses are allocated to nodes When a node is added, the load balancing service obtains an idle IP address from the IP address pool and allocates it to the newly added node. If no idle IP address is available, the load balancing service determines whether any existing clustered node has multiple IP addresses. If yes, the load balancing service deprives the clustered node of one IP address and allocates it to the newly added node, ensuring that the new node takes part in load balancing. If no, an alarm is generated asking the OceanStor 9000 system administrator to add new idle IP addresses to the IP address pool. The following figure shows a schematic diagram of how IP addresses are allocated to newly added nodes. Issue 01 (2014-06-20) Huawei Confidential 7
Figure 3-3 Schematic diagram of how IP addresses are allocated to newly added nodes If some of the network adapters equipped on a node fail, which causes an IP address problem, an IP address failover is implemented within the node to switch IP addresses from the failed network adapters to functional network adapters. If a node has multiple network adapters, IP addresses are evenly allocated to them. If a node fails, the node with the lightest load in the cluster is selected to take over, as shown in the following figure. Figure 3-4 Schematic diagram of an IP address switchover when a node fails In the event that a failed node recovers, the load balancing service obtains an idle IP address from the IP address pool and allocates it to the recovered node. If no idle IP address is available, the load balancing service determines whether any existing clustered node has multiple IP addresses. If yes, the load balancing service deprives the clustered node of one IP address and assigns it to the recovered node. If no, an alarm is generated asking the OceanStor 9000 system administrator to add new idle IP addresses to the IP address pool. The following figure shows a schematic diagram of IP address switchover in the event that a node recovers. Issue 01 (2014-06-20) Huawei Confidential 8
Figure 3-5 Schematic diagram of an IP address switchover when a node recovers 3.3 Diverse Load Balancing Policies The OceanStor 9000 load balancing service supports the following load balancing policies, which can be configured by users based on site requirements: Round robin (the default load balancing policy) Nodes are selected in sequence to process client connection requests. A three-node environment is used as an example, as shown in the following figure. Tn indicates the time of a DNS request. The load balancing service working in round robin mode selects node 1 at the T1 point in time to process services, node 2 at the T2 point in time to process services, and node 3 at the T3 point in time to process services. Figure 3-6 Schematic diagram of load balancing in round robin mode CPU usage The node with the lowest CPU usage is selected to process client connection requests. A three-node environment is used as an example, as shown in the following figure. Tn indicates the time of a DNS request. The load balancing service working based on CPU usage selects node 1 at the T1 point in time to process services, node 2 at the T2 point in time to process services, and node 3 at the T3 point in time to process services. Issue 01 (2014-06-20) Huawei Confidential 9
Figure 3-7 Schematic diagram of load balancing based on CPU usage Number of connections The node with the least NAS service connections is selected to process client connection requests. A three-node environment is used as an example, as shown in the following figure. Tn indicates the time of a DNS request. The load balancing service working based on the number of connections selects node 1 at the T1 point in time to process services, node 2 at the T2 point in time to process services, and node 1 at the T3 point in time to process services. Figure 3-8 Schematic diagram of load balancing based on the number of connections Node throughput The node with the lowest network throughput is selected to process client connection requests. A three-node environment is used as an example, as shown in the following figure. Tn indicates the time of a DNS request. The load balancing service working based on the node throughput selects node 2 at the T1 point in time to process services, node 1 at the T2 point in time to process services, and node 1 at the T3 point in time to process services. Issue 01 (2014-06-20) Huawei Confidential 10
Figure 3-9 Schematic diagram of load balancing based on node throughput Node capability The load balancing service calculates the static capability value of a node based on the node's hardware configurations such as the CPU and bandwidth. The load balancing service determines the dynamic load status of a node based on the current running status of the node. The capability of a node is co-determined by the static capability value and the dynamic load status. If the load of a node is heavy, the capability value of the node decreases. If the load of a node is light, the capability value of the node increases. Nodes are selected to process client connection requests based on their capability values. A node with a larger capability value is more likely to be selected. If a node has multiple IP addresses, the IP address with a larger capability value is selected first. The major parameters are calculated as follows: Static capability value of a node Static capability value of a node = Number of CPU cores x CPU dominant frequency x (100% + Memory size/100 GB) x Network adapter bandwidth Network adapter bandwidth = Min (maximum bandwidth of a NAS service port: 20 Gbit/s) Load of a node Load of a node = Max (CPU usage, network adapter usage) Static capability value of an IP address Static capability value of an IP address = Maximum bandwidth of the physical port to which the IP address is allocated/number of IP addresses configured Load of an IP address Load of an IP address = (TX bandwidth of the physical port to which the IP address is allocated + RX bandwidth of this physical port)/(maximum bandwidth of this physical port to which the IP address is allocated x 2) A three-node environment is used as an example, as shown in the following figure. The minimum capability value among all nodes is used as the basic capability value. In each node selection, if the capability value of the current node is larger than the basic capability value, the current node is selected and the capacity value is updated (Capability value of a node = Original capability value Basic capability value). Otherwise, the system compares the capacity value of another node with the basic capability value until an appropriate node is selected. If the capability values of all nodes are smaller than the basic capability value, the capability values of all nodes are set to the initial value, and then a node is selected again. Issue 01 (2014-06-20) Huawei Confidential 11
As shown in the following figure, of the six service requests initiated in one load balancing period, node 1 has processed three, node 2 has processed two, and node 3 that has the smallest capability value has processed only one. Node 1 that has the highest capability value has processed the most service requests. This is load balancing achieved based on capability value policies. After the T6 point in time, the capability value of each node will be updated, and the next load balancing period starts. Figure 3-10 Schematic diagram of load balancing based on node capacity values 3.4 Zone-based Node Management The OceanStor 9000 supports a maximum of 288 nodes in 32 zones. The load balancing service allows IT administrators to allocate nodes to different zones for easy management. An independent load balancing policy and an independent domain name can be configured for each zone. A common practice is to set a high-performance zone and a high-capacity zone, allocate nodes of specific capabilities to the two zones, and configure an independent domain name for each zone. Users use different domain names to access different zones. As shown in the following figure, four nodes are allocated to two zones. Domain name highperformance.9000.com is configured for the high-performance zone, and highcapactiy.9000.com is configured for the high-capacity zone. Figure 3-11 Schematic diagram of node zone access Client Access by highperformance.9000.com Access by highcapactiy.9000.com Storage system High-performance zone High-capacity zone Node 1 Node 2 Node 3 Node 4 Issue 01 (2014-06-20) Huawei Confidential 12
4 Customer Benefits The OceanStor 9000 load balancing service supports intelligent client connection management, load balancing, and failover, improving system availability and ensuring high performance. The OceanStor 9000 load balancing service provides the following benefits: Provides unified domain name access to simplify connection to the OceanStor 9000. The OceanStor 9000 is presented to users as a single network unit. Changes to the OceanStor 9000 are transparent to users. Provides diverse load balancing policies that enable client connection requests to be evenly distributed to all OceanStor 9000 nodes, preventing performance bottlenecks. Provides services of different levels to different users through zone-based management and load balancing policies. Provides highly available and reliable connection management based on IP address failover and failback. If a node of the OceanStor 9000 fails, users that have been connected to the node can still access the OceanStor 9000. Provides intelligent IP address management. When a node is added, an IP address is automatically allocated to it. When a node quits, its IP address is automatically migrated. In this way, changes to node quantity are transparent to user connection, but performance improvement can be sensed. Issue 01 (2014-06-20) Huawei Confidential 13
5 Acronyms and Abbreviations DNS NAS domain name system network attached storage Issue 01 (2014-06-20) Huawei Confidential 14