Building HA Linux Cluster A tutorial for IEEE Cluster Conference 2001

Transcription

1 Building HA Linux Cluster A tutorial for IEEE Cluster Conference 2001 Ibrahim.Haddad@Ericsson.com Ericsson Research Corporate Unit Ericsson Canada Purpose of the tutorial 1. Share our experience in building near-telecom grade HA Linux clusters. 2. Address design and implementation issues. 3. Share our vision of the future of carrier class server nodes Ericsson Canada 1

2 Tutorial Description Introduction Clusters and cluster Computing Types of clusters HA clusters System HW architecture The ARIES project Linux: An OS for telecom platforms Intermediate Issues in building HA clusters Steps ( ) Testing the clusters availability and performance Advanced topics Load balancing and traffic distribution IPv6 Security Cluster simulation Ericsson Canada Clustering and Ericsson Research Open Architecture Research Mission Conduct applied research for the access, application, network and services for advanced mobile network, in co-operation with product units, core units, customers and research organizations to demonstrate the viability of distributed system for Servers and Applications in IP Network. Activities Cluster computing, cluster simulation, IP telephony services, IP routers ECUR Lab State of the art telecom-grade equipment Ericsson Canada 2

3 Part I: Introduction to Clusters Clusters HA Cluster High availability Fault tolerance Overview of the system architecture Ericsson Canada What is a cluster? A cluster is a collection of connected, independent computers that work together to solve a problem Ericsson Research Open Architecture Lab Montreal - Canada Ericsson Canada 3

4 Clustering Goals High Availability Isolate or reduce the impact of a failure in the machine, resources, or device through redundancy and fail over techniques. Scalability Expand the capacity of servers in terms of processors, memory, storage, or other resources to support business growth Improved processing speed High performance / Improved access time / Improved response time Load Balancing Efficient resource utilization Manageability Reduce system management costs through appropriate system management facilities Ericsson Canada Where are clusters used today? Clusters are used in: Computational science research Universities National labs ISPs and ASPs Telecom research labs / telephony services research labs Government labs (such as Ministers of Defence ) However, they still are: Too hard to set-up and use Too many options without clear winners Too many learning curves to climb too many times Ericsson Canada 4

5 HA Clusters Ericsson Canada HA Clusters HA clusters have the ability to continue operations even if a server fails HA systems are not anymore regarded for traditional mission critical applications. An increasing need for almost all commercial applications and systems to provide users with minimal downtime. This even extends, for example, to customers demanding minimal downtime for regular system maintenance/upgrades. Cost of downtime Ericsson Canada 5

6 High Availability (HA) HA is defined as the capability of a computing resource to remain on-line in the face of a variety of potential subsystem failures. Failure of a power supply, network access, storage devices... Routine maintenance to upgrade an operating system or application may demand that a subsystem be taken off-line. High availability is measured in percentage In a monolithic system (vs. a cluster), each of these events would interrupt service Ericsson Canada HA Degrees Ericsson Canada 6

7 Fault tolerance An alternative approach to gaining greater availability is a fault tolerant architecture. All critical subsystems in a fault tolerant system are redundant In the case of a subsystem failure, a "hot spare" is available Replicated power supplies, cooling systems, disks, CPUs, The fault tolerant approach is expensive. To guarantee reliability, you essentially purchase several computing subsystems, only one of which carries the workload at any one time. Second and subsequent systems shadow processing and mirror the storage of data, without contributing to the overall capacity of the system Ericsson Canada High availability vs. fault tolerance High availability for cluster architectures works differently. 5 clustered nodes may divide the load for a set of critical applications. All 5 nodes contribute toward processing the tasks at hand. If 1 of the nodes fail, then the 4 remaining nodes pick up the load. Depending on the load at the time of the failure, performance will drop no more than 20% (1/5). Switching the load of the failed machine to other machines usually takes a short period of time Ericsson Canada 7

8 High availability vs. fault tolerance Monolithic systems can be expected to perform 99% of the time. 1% downtime translates to 90 hours in a year, or about 3.5 days. Not acceptable in the telecom world Fault-tolerant systems can improve reliability to %. At this level of system reliability, it is far more likely that extrinsic factors will interrupt service. Cluster architectures can be tuned in accordance with the cost of downtime Ericsson Canada Software vs. Hardware availability Hardware availability is increasing at a higher paste than software availability/reliability In some cases, the platform may be available but not the application Software has bugs; it may cause applications to crash. Keeping redundancy in applications and maintaining processes state is complex. In telecom, the required uptime includes both platform and applications uptime End-users don t care about running platforms when the required application is unavailable Ericsson Canada 8

9 Failover Failover is the ability of a cluster to detect problems in a node and to accommodate ongoing processing by routing applications to other nodes. This process may be programmed or scripted so that steps are taken automatically without operator intervention. Fundamental to failover is communication among nodes signaling that they are functioning correctly and reporting problems when they occur Ericsson Canada Heartbeat Each node listens actively to make sure that all of its companions are alive. When a node fails, cluster interconnect software takes action. More sophisticated cluster software reacts to the problem by: Shifting applications and users automatically, and Reconnecting to one or more healthy nodes of the cluster. Journals may be necessary to bring an application up to its current transaction with integrity. Databases may need to be reloaded. The catastrophic failure of a node is one event to which the cluster management environment must respond. Other potential problems may be a failure of a network interface, RAID failure, Ericsson Canada 9

10 Issues in Building HA Clusters How to automatically build and boot the nodes? Which file systems to use in the cluster? What types of traffic distribution and load balancing mechanisms to adopt depending on the applications? How to build redundancy and to which extend? How to manage the cluster, remotely? How to add/remove nodes without affecting the operations? Ericsson Canada The HW System Architecture A typical research Linux cluster for telecom applications at Ericsson Research Canada Ericsson Canada 10

11 The Needs Multi-CPU rackmount server systems that are flexible can support several developers working on different projects instead of having one server shelf and many diskless CPUs, we wanted shelves that can be autonomous from each other The systems should be as close as possible to real production systems so that system integration problems can be studied and overcome 48 volts power supply systems CompactFlash support pre-installed SCSI disks and tape backups Software RAID configurations as many fast Ethernet ports as possible on each CPU Ericsson Canada Server hardware platform 41U 40U 31.5U 23U 22U 21U 18U Compact PCI design -48V Central Office Powered NEBS Compliant Ready 16 P3 500MHz 512 Mbytes RAM 6 Ethernet Ports / Processor 8 SCSI Disk banks (3x18GBytes) Fully Redundant and hot swap Ericsson Canada 11

12 The solution Each system has four shelves (a total of 16 CPUs and 24 hard drives) 4 Pentium III 500MHz CPUs, each with 512MB RAM, CompactFlash, 2 onboard Fast Ethernet ports and 4 Fast Ethernet ports supplied by a Znyx 474 card two of the CPUs are diskless, two have a SCSI bus with three 18GB hard drives and a DDS-4 capable DAT tape drive Each system also has a Knurr alarm management system and two 37-port (36 100baseT + one fiber) 3com switches Ericsson Canada IO Front View 4x 150W PS in n+1 Hot swappable Dual Dat tape Hot swappable Dual raid array 3 18GB SCSI each All hot swappable Technician GND for wrist strap Chassis GND Quad cpci-mxs64 processors with a quad Ethernet Znyx card all hot swappable Hot swappable Ventilation: In - front bottom Out: rear top Ericsson Canada 12

13 IO Rear View All rear I/O cpci-mxs64 and quad Ethernet Znyx cards Sealed for airflow and EMI protection Dual input Power Filter module (single input for this first iteration) -48V input Alarm information output Wiring duct Ericsson Canada Processors Magazine Front view 6x 150W PS in n+1 Hot swappable 15 x cpci-mxp64 processors, all hot swappable Technician GND for wrist strap Hot swappable Ventilation: In - front bottom Out - rear top Ericsson Canada 13

14 Alternative (cheap) hardware used to build clusters Off-the-shelf 1U Celeron/Pentium III 256/512/768 MB RAM 20 GB IDE HD Floppy/CD-ROM 2 fast Ethernet ports 2 USB ports Cybex remote control unit 48 port Ethernet Switch Fiber connections Ericsson Canada Part II: The ARIES Project Advanced Research on Internet E-Servers Ericsson Canada 14

15 Overview Today vs. tomorrow Challenges ARIES Project Internet Servers Requirements Operating System Requirements Linux Kernel ARIES research areas Ericsson Canada Today vs. tomorrow 32 bit architecture Memory < 1 GB Processor 1 GHz Fast Ethernet IPv4 Network File Systems TelORB Basic Security Hundreds of TPS < 2 Million Subscribers 64 bit architecture Memory > 10 GB Processor > 10 GHz Gigabit Ethernet IPv4 and IPv6 Storage Area Networks TSP (TelORB & Linux) Severe Security Reqs. Thousands of TPS Subscribers > 10M Ericsson Canada 15

16 Challenges Bringing telecom-grade characteristics to Linux HA, scalability, Interoperability, HW, languages Supporting IPv4 and IPv6 Achieving linear scalability Reliable distributed file systems and distributed network storage Efficient load balancing and traffic distribution Providing a high level of security Providing remote operation & management Ericsson Canada Internet Servers Requirements Capacity Scalability Scale up any of the components in order to achieve a linear increase. Load Balancing Provides dynamic load-balancing mechanisms. The load balancing mechanism detects and reacts to the unavailability, addition and removal of components in the system. Availability Meet high availability requirements Operation and maintenance Performed (remotely) without affecting the system performance and availability Response Time Ericsson Canada 16

17 Internet Servers Requirements Geographical Diversity Spread across several Points of Presence Support geographic mirroring The communication is over dedicated high-speed links Protocols Supports UDP, TCP, HTTP, H.323, IIOP (CORBA), and SIP in the same scalable fashion Single IP Interface Clients access the server application through a single IP address Security High security requirements (e- and m-commerce) Ericsson Canada Operating System Requirements Characteristics 1. Very high availability and robustness implemented in software 2. (Soft) Real Time 3. Scalability 4. Performance Openness 1. Hardware 2. Languages: C++, Java 3. Interoperability: CORBA/IIOP, TCP/IP, Java RMI 4. 3rd party Software Ericsson Canada 17

18 ARIES 2000 Find and prototype the necessary technology to prove the feasibility of an Internet Server that provides telecom-grade characteristics: High Reliability ( % uptime) High Performance (x number of transactions/sec/processor) High Scalability (Number of CPUs and IP connections) High Throughput (fast and reliable voice/data streaming) Open to Fast Reconfiguration Use Linux and Open Source Software as the base technology Use commodity off-the-shelf hardware and software Better cost/performance ($/TPS) ratio Ericsson Canada ARIES 2001 Find, prototype and develop the necessary technology, which will enhance the clustering capabilities of TelORB and Linux to fulfil the future demands for Mobile Internet Servers. Research Areas: IPv6 Security Load Balancing Bringing IPv6 to carrier-class server nodes Build security at various cluster levels Efficient use of cluster resources Ericsson Canada 18

19 Part III: Linux as a candidate operating system for a near-telecom grade clusters Ericsson Canada Kernel Features Openness (HW/SW, Languages, Interoperability, 3 rd party SW) Flexibility (access to kernel code) Value for money Kernel Stability Optimizable Kernel (for size, speed and features) Offers support for embedded devices (PDAs, cell phones ) IPv6 Support Access to faster releases of IP features than other operating systems Supports a variety of network protocols Ericsson Canada 19

20 The Enterprise Linux Kernel (v 2.4) New System Architecture Features Support for multiple architectures inc. IA-64 (not bound to 1 supplier) Improved support for symmetric multiprocessing (SMP), up to 64 processors Enhanced task synchronization and threading facilities for increased efficiency File System Features Improved Virtual File System layer for efficiency Support advanced file system capabilities such as journaling ReiserFS, ext3, xfs, and jfs Ericsson Canada The Enterprise Linux Kernel (v 2.4)... Improved Networking Features Rewritten networking layer resulting in performance improvements Improved IPv4 implementation / Improved NAT performance Improved Support for IPv6 Kernel HTTP Daemon New version of NFS (enhanced security and performance) Support for Logical Volume Management Increased Device Support Support for 16 Ethernet cards Support for new SCSI controllers Support for additional RAID devices Ericsson Canada 20

21 Break Next: Building the Linux HA Cluster Ericsson Canada Part IV: Building a HA Linux Cluster How can we build a Linux based cluster using Open Source software to achieve continuous availability? Part of the work that was conducted as part of ARIES Ericsson Canada 21

22 Outline Architecture of a typical cluster Basic steps into building the cluster Automating installations Building the disks Automating installations Lessons learned Installing and starting application servers Building Redundancy Traffic distribution with LVS Load balancing with MOSIX Ericsson Canada Architecture of our typical Linux cluster 2 Redundant master nodes DHCP server TFTP server NFS server Install/BpBatch Server Security gateway NTP server Single IP Interface Incoming Traffic Single IP Interface Master Nodes Traffic Nodes Traffic nodes Host application servers (Apache, Jigsaw, Tomcat, Real Server, Icecast) Serve requests Ericsson Canada 22

23 Basic steps Prepare the master nodes Install Linux Red Hat 6.2 server installation Configure Linux kernel to support required configs Very minimal and optimized Configure the network settings 4 ports, two of each are on different networks Setup Ethernet redundancy Setup DHCP Collect nodes information such as MAC addresses Setup NFS Patch the kernel source code with NFS redundancy patches Recompile the kernel and the special mount program the mount point and /etc/exports on both servers are the same Setup TFTP Setup NTP Install BpBatch Ericsson Canada Basic steps Kernel and ramdisk preparation Build a Linux kernel to be used on the traffic nodes 2 kernels should be prepared depending whether the node has disks or not Prepare 2 types of ramdisks one for nodes with disks and one for diskless nodes Prepare traffic nodes Modify the BIOS setup so that all the boot sequence is as follows: For nodes with disk: LAN 1, LAN 2, Flash, disk For diskless nodes: LAN 1, LAN2, Flash (For disk-processors, check for flag file in Flash to where to boot from) Ericsson Canada 23

24 Cluster server automated installation In each cabinet, two redundant CPUs with disks are Linux boot servers for the rest of the cabinet. In the Linux cabinet, the remaining six CPUs with disks can have their configuration rebuilt automatically from the two master CPUs. Automated installation of these systems using BpBatch The main feature of BpBatch is the partition cloning facility, which let us create an image of a computer's hard disk partition and then distribute and install this image on a cluster of PC. Big success: we can do a complete rebuild of a RAID system from a corrupted base (all disks corrupted and CompactFlash corrupted) entirely from the network Ericsson Canada Sample BpBatch script diskless nodes For diskless processors: Logvars "Print" Print "INFO: Starting BpBatch script\r\n" Set CacheNever="ON" LinuxBoot "LATEST-bzImage" "root=/dev/ram "ramdisk_size =50000" "LATEST-ramdisk" Ericsson Canada 24

25 Sample BpBatch script disk nodes LogVars "Print" Print "INFO: Starting BpBatch script\r\n" if valid 0:1 goto ChooseStage Print "INFO: CFlash is not valid\r\n" goto RebuildDisk :ChooseStage Print "INFO: CFlash is valid\r\n" if exist "{1:1}/second.step" goto Stage2 Print "INFO: Reboot from disk sdc\r\n" if exist "{0:1}/third.stp" goto Stage3 Print "INFO: Booting raid from disk sda\r\n" if exist "{0:1}/boota.stp" goto JustBoota Print "INFO: No I mean Booting raid from disk sdb\r\n" if exist "{0:1}/bootb.stp" goto JustBootb Print "INFO: Oups! System in bad shape, we have to Rebuild\r\n" goto RebuildDisk Ericsson Canada Sample BpBatch script master nodes At any time, a master node can be built given that the other master node is up Prepared a minimal ramdisk and a script that would help us build a disk image very fast. Boot from LAN, configure network setting based on DHCP broadcast from the alive master node, rebuild disk, install all data, start services Ericsson Canada 25

26 Automated Installations Disk-CPU Automatic Rebuild: Boot from LAN (LAN 1/LAN 2) If LANs are not available boot from Flash If LANs and Flash not available, boot from disk Activate the Ethernet connections We have 6 connections, we only activate 4 Build a disk CPU from scratch Using BpBatch scripts Setup RAID Next slide Install ramdisk which include all the applications Start application servers such as web and streaming servers Ericsson Canada Automatic RAID Setup Check flash for a special file If file does not exist, do a boot from LAN and rebuild disk w/ RAID If file exists, then do a boot from disk Stage 1 Boot from LAN Stage 2 Boot from LAN on SDC Stage 3 Add SDC to RAID setup Disk Map S D A Flash S D B S D C Install minimal kernel Boot from LAN on SDA Rebuild Flash Partition SDC Download ramdisk and install it on SDC Partition SDA and SDB for RAID Mark SDC as faulty during RAID setup Perform RAID setup Boot from LAN on SDA System is up Synchronize disks Ericsson Canada 26

27 Diskless CPU Automatic Rebuild Disk-CPU Automatic Rebuild Boot from LAN (LAN 1/LAN 2) Activate (4) Ethernet connections Download ramdisk Start application servers Config files are stored on NFS to keep ramdisk size to minimal Ericsson Canada Automatic installations evaluation Disk-CPU automatic installation: Advantages: No manual intervention. The setup can be easily adapted to different configurations. Very fast to sub-cluster Inconveniences: Not very fast the RAID setup consumes 3 reboots Limited ramdisk size due to BpBatch and TFTP limitations Diskless CPUs automatic installation: Advantages: Very fast (less than 45 secs.) Inconveniences: Limited ramdisk size (15 MB) due to BpBatch and TFTP limitations Ericsson Canada 27

28 Building the Ramdisk Ramdisk for Diskless nodes Start from someone else s ramdisk; many available of the Internet Prepare your own Do a very minimal install for a system (just a the minimum required to have a working system) Get rid of all the stuff that you don t need Write a script to build the ramdisk image Save the ramdisk image under the tftpboot directory Ramdisk for disk nodes Build and customize one node Use BpBatch utilities to capture the disk image Use BpBatch scripts invoked by DHCP to rebuild the disks Ericsson Canada Things learned about the freeware BpBatch Complete rebuild of a SCSI RAID configuration through BpBatch requires three reboot sequences and writing to the CompactFlash Handling of disk Master Boot Records (MBRs) is buggy Had to learn how to work around BpBatch bugs, took a lot of time Few types of disk partitions are supported access to Linux EXT2 and DOS FAT16 partitions works access to DOS FAT12 (the default CompactFlash setup) or Linux RAID partitions does not work in general, BpBatch does not deal well with partition tables it has not created itself Use of DHCP-supplied values to drive BpBatch is also somewhat buggy Ericsson Canada 28

29 More things learned about auto-install BpBatch only knows how to transfer things using unicast TFTP this is somewhat unreliable and introduces unnecessary limits on the file sizes which can be downloaded using BpBatch A network installation sequence that requires multiple reboots must keep state in the client, since the boot server must be stateless Since BpBatch will not read RAID partitions, this forced us to keep the boot stage information in the CompactFlash Conclusion: Find alternatives to BpBatch to reduce bug-hunting and system rebuild time Commercial version of BpBatch: Ericsson Canada Application servers As test application servers, we used: Web Servers: Apache, Jigsaw, and Tomcat Streaming servers: Real Server and Icecast Binaries are contained within the ramdisk /usr/local/bin/ Configuration files are the same for all CPUS One copy is available on NFS and shared among all CPUs /nfs/apache/httpd.conf for instance Update one copy only and you get a new config on all nodes Startup script are in /etc/rc.d/* Jigsaw and Tomcat rely on JAVA JDK was available on NFS for diskless nodes due to space limitations imposed on the size of the ramdisk /nfs/jdk1.2.2/ Ericsson Canada 29

30 Redundancy in a clustered system: How to build a cluster that can meet our HA requirements? Ericsson Canada Redundancy Schemes Servers must maintain high availability. How to reach a telecom-grade uptime? Building redundancy into the system Redundancy at the Network Level Redundancy at the File System Level Redundancy at the Disk Level Redundancy at the CPU Level Ericsson Canada 30

31 Redundancy levels Ethernet Connections Monitor status of link1 If link1 is down: Delete route to link1 Direct traffic to link2 NFS Redundancy Implemented at kernel level Special mount program Switch Servers on timeout Software RAID Automatic Setup RAID 1 and RAID 5 Disk Architecture Independent eth2 eth3 Server 1 Primary NFS Server Internet eth2 eth3 Server N eth0 eth1 eth0 eth1 NFS Data Source 1 Firewall Cluster Single Entry Point Secondary NFS Server NFS Data Source Ericsson Canada Ethernet connections Ericsson Canada 31

32 Ethernet redundancy Goal: Keep Ethernet connections up The Ethernet redundancy daemon monitors the link status of the primary port. On link down, the route of the first port is deleted the traffic goes to the second port When the link goes up again, wait a bit, and switch to the primary port. The backup port routes have metrics bigger than the primary one. Reads router data from /proc/net/route Deletes routes bound to port eth[n+1] Adds routes for port eth[n+1] using configuration for port eth[n] with a higher routing metric value (i.e. lower priority) Ericsson Canada Ethernet redundancy daemon operation phase Poll running bit (patched tulip driver) for port eth[n] and if down then delete routes bound to eth[n] Traffic bound to port eth[n] is now diverted to port eth[n+1] Poll running bit for port eth[n] and if up then re-install routes for port eth[n] and ping to broadcast address. Traffic bound to port eth[n+1] is now diverted to port eth[n] with the lower metric value (higher routing priority) Ericsson Canada 32

33 Ethernet redundancy daemon Configure a pair of ports eth[n] and eth[n+1] with the same IP and MAC addresses Use the following command to change MAC address of a network interface card: ifconfig eth[n+1] hw ethr aa:bb::cc:dd:ee:ff Run ERD daemon Ericsson Canada NFS redundancy Goal: Maintain data availability by having redundant NFS servers NFS kernel files have been modified to support NFS redundancy. Kernel File Location /usr/src/linux/fs/nfs/inode.c /usr/src/linux/net/sunrpc/sched.c /usr/src/linux/net/sunrpc/clnt.c Changes Description All the NFS redundancy support Raise the timeout flag when there is one Raise the timeout flag when there is one Ericsson Canada 33

34 NFS redundancy A special version of the mount program has also been implemented. The 2 servers are passed to the mount program and then to the kernel. % mount t nfs server1,server2:/nfs_mnt_point The nfs mount point has to be the same on both servers When a timeout occurs: A kernel thread is waken up. The data from the 2 servers is switched. The different caches are cleaned up. The dcache is updated with new handles from the new server Ericsson Canada Data redundancy through software RAID Goal: To have data available in the case of a disk crash Solution: Use software RAID 1 and RAID 5 The first thing we did, before working on the boot sequence, is installing and setting up 1 CPU with raid partition using the cdrom installation with RedHat 6.2 (pro. version) With this installation, we tar and zip the different partitions we have (next slide) Ericsson Canada 34

35 Disks partitioning sda1, sdb1 and sdc1 are representing the md0 partition that is /boot sda6, sdb6 and sdc6 are representing the md1 partition that is /var sda5, sdb5 and sdc5 are representing the md2 partition that is / sda8, sdb8 and sdc8 are representing the md3 partition that is /home sda7, sdb7 and sdc7 are representing the md4 partition that is the swap Ericsson Canada Building RAID setups We wrote a script that auto-format a disk with this exact geometry and partitions. You need to compile a kernel with these options in block devices : Autodetect RAID partitions [y] Linear (append) mode [m] RAID-1 (mirroring) mode [m] RAID-4/RAID-5 [m] Initial RAM disk (initrd) support [y] To automate the setup on other nodes, we used DHCP with BpBatch scripts. With bpbatch, we download a small kernel image that will be used to partition the disk with the script we did Then, the CPU downloads via ftp the full kernel, the different files systems, the config files and scripts we are going to use on next step. (check the Automatic RAID Setup slide) Ericsson Canada 35

36 Testing the achieved availability Ethernet Disconnect Ethernet cables while running a benchmark No interruption in service 0 failed requests SW RAID Remove disk from cabinet, wait few minutes, put it back The disk will be detected and will be synchronized with the other disks Synchronization time depends on the size of the disk and the number of changes NFS Shutdown 1 NFS server or disconnect its Ethernet cables Application servers switch to use the second NFS server Transparent No fails Ericsson Canada Screen shot of cpu Ericsson Canada 36

37 Traffic Distribution: How can we allow an infinite amount of clients to reach an infinite amount of servers presented as a single virtual IP address? Ericsson Canada Ideal solution High availability No single point of failure Minimal impact in case of failure Scalability same cost/performance ($/TPS) same response time No impact/requirements on clients/servers/routers Ericsson Canada 37

38 Linux Virtual Server LVS is a server built on a cluster of real servers, with the load balancer running on Linux. 3 architectures: Network address translation (NAT), direct routing (DR), IP tunnelling (IPT) The architecture of the cluster is transparent to end users Users only see a single IP address Detects failure of a node and reconfigure LVS is not highly scalable Bottlenecks at LVS Server Level Limited Scalability at Nodes Level (~20) Ericsson Canada LVS via NAT Ericsson Canada 38

39 Scalability of LVS via NAT 8 traffic processors getting their traffic from WebBench 8 traffic processors getting their traffic through LVS Requests / Second Requests/Second _client 4_clients 1_client 8_clients 16_clients LVS Results 8_clients 12_clients 16_clients 20_clients 24_clients LVS vs Non-LVS Setup (1 LVS/8 Traffic CPUs) 24_clients 32_clients 40_clients 48_clients 56_clients 28_clients 32_clients 36_clients 40_clients 44_clients 48_clients 52_clients 56_clients 60_clients Non-LVS Setup LVS Setup Ericsson Canada LVS via DR Ericsson Canada 39

40 Performance comparison: DR vs. NAT 8 traffic processors getting their traffic through LVS using Direct Routing 8 traffic processors getting their traffic through LVS using NAT DR is almost 3x more performant Requests per Second _client 8_clients 16_clients LVS: DR vs NAT 24_clients 32_clients 40_clients 48_clients 56_clients LVS-DR LVS-NAT Ericsson Canada LVS conclusions LVS is easy to install and manage, and very useful. LVS is a potential solution for small to mid-size web farms that need a software-based solution for traffic distribution. LVS's future is promising with the determination to add more: load-balancing algorithms geographic-based scheduling the integration the heartbeat code and the CODA distributed fault-tolerant file system into the virtual. LVS's implement the virtual server in IPv6. Compared to other packages: LVS provides many unique features such as the support for multiple scheduling algorithms, and the support for various methods for requests forwarding (NAT, direct routing, tunneling). Our next step regarding LVS is to try out the other two implementations (direct routing and IP tunneling) and compare the performance with the NAT implementation on the same setup. We found some restrictions using LVS under heavy load. LVS can not handle the high number of transactions per second that our servers will be receiving Ericsson Canada 40