Load Balancing
Topics 1. What is load balancing? 2. Load balancing techniques 3. Load balancing strategies 4. Sessions 5. Elastic load balancing
What is load balancing? load balancing is a technique to distribute workload evenly across two or more computers, network links, CPUs, hard drives, or other resources, in order to get optimal resource utilization, maximize throughput, minimize response time, and avoid overload. Wikipedia
Load Balanced Servers Internet Server 1 Server 2 Server N
Load Balancing + Failover
Load Balancing + Failover + Cloud
Round robin DNS DNS global server load balancing Layer 4 load balancer Web switch Reverse proxy Load balancing techniques
Round Robin DNS Configure multiple A records for single name Return next IP address from circular list Some servers may try to return closest IP address. Problems Clients cache addresses. No awareness of availability status of addresses.
> dig google.com Round Robin DNS Entry ;; ANSWER SECTION: google.com. 21 IN A 74.125.225.18 google.com. 21 IN A 74.125.225.19 google.com. 21 IN A 74.125.225.20 google.com. 21 IN A 74.125.225.16 google.com. 21 IN A 74.125.225.17
DNS Global Server Load Balancing 1. Client makes a DNS query to ISP DNS server. 2. ISP DNS server looks up DNS records. 3. GSLB receives DNS query. 4. GSLB algorithm examines site health and latency to determine which site best. 5. GSLB returns IP address to client of site that is available and has lowest latency. Problem: DNS servers and clients cache responses for long periods of time.
Application Layer 4 Load Balancing Switch Presentation Session Network Address Translation (NAT) for LB One external IP address Switch maps incoming connections to one of the pool of servers. Switch maps outgoing connection source IP address to IP address of the switch. Additional features Better load balancing algorithms Detection of availability status of servers Transport Network Data Link Physical
Application Web Switch AKA content switch or application switch Presentation Session Transport Network Data Link Physical Balances load at application layer Multiple load balancing algorithms Handles SSL for servers May also perform compression May also add firewall features
http://www.f5.com/pdf/products/big-ip-platforms-ds.pdf
Reverse Proxy Proxy server has network identity Forwards requests to backend web servers Features can include caching, compression, SSL
Open Source Load Balancers Perlbal used by LiveJournal, TypePad HAProxy used inside hw + sw load balancers Varnish HTTP accelerator, caching + balancing Pound Security-focused HTTP accelerator mod_proxy_balance Apache module NGinx lightweight, high performance web proxy
Round robin Least connections/time Predictive Random Weighted strategies Load balancing strategies
Round Robin Server 1 Server 5 Server 2 Server 4 Server 3
Least Connections / Response Time Send requests to server with the least number of connections or lowest response time. Good for balancing between requests with different requirements or servers with different performance levels. Problems can arise with multiple load balancers making decisions in parallel. Weights can be added for manual tweaking.
Predictive Round robin or least connections with additional heuristics to compensate for information staleness issue arising from many short rapid transactions.
Random Either select server at random or combine with resource-based algorithm to deal with information staleness problems. Weighted random adds manual constant to probability of choosing particular servers. A dual-core isn t twice as fast as a single CPU. Requires trial and error experimentation.
Sessions
The Problem User interactions with a server have a certain amount of state. Authentication Current stage of transaction Shopping carts, etc. How can the application preserve state when the load balancer sends next request to a different server?
Session Stickiness Stickiness requires session-aware load balancer Ensures that future requests from same session always go to same server. Problems Lack of failover since requests tied to single server Difficult to allocate resources effectively since sessions vary tremendously in duration and size
Client-side State Why not store all state in client cookies? Insufficient client storage (ameliorated by HTML5) Insecure (can t trust client to control prices, id) Solution Store frequently accessed low security data in client cookies. Store rarely accessed data in central backend storage and take performance hit to access when necessary.
Elastic load balancing
Elastic Load Balancing Distributes traffic across EC2 instances Instances can be located across multiple AZs Supports any TCP based protocol Monitors instance health and will not distribute traffic to unhealthy instances Supports SSL termination Supports session stickiness Provides metrics to CloudWatch
ELB Algorithms 1. Weighted round robin Sends new request to instance handling smallest number of requests. Round robin choice if multiple instances have same smallest number of requests. Can configure health checks to prevent ELB from sending requests to unresponsive instances. 2. Session sticky Weighted round robin, but all requests that are part of a session go to the same server.
ELB Lifecycle 1. Create an ELB List of AZs Parameters for health check List of listeners 2. Add instances to ELB By instance ID ELB will track status: InService, OutOfService 3. Advertise public DNS name of ELB 4. Modify number of instances to match traffic Or setup CloudWatch/AutoScale to handle for you 5. Delete ELB when unneeded
ELB, CloudWatch, and AutoScale
CloudWatch Monitoring service for EC2 CPU utilization, Data transfer, Storage usage Pricing Basic Monitoring with 5 minute granularity free Detailed Monitoring (1 minute) for 1.5 per hour 10 per alarm after first 10 alarms
CloudWatch Terminology A namespace represents a source of data AWS/EC2 AWS/ELB A measure is a raw, observed data value One minute s worth of observation A unit is an attribute of a measure Seconds, %, bytes, bits, counts, bytes/s, bits/s A dimension is a refined view of a type of data AvailabilityZone, ImageType, InstanceID, A metric is a stored, processed measure A statistic is a computed attribute of a metric Minimum, maximum, average, sum
AutoScaling Group AutoScaling Set of EC2 instances that should scale together Triggers Scale on CloudWatch alerts Scale on time-based schedule Fixed number of healthy instances Examples Add 3 instances if CPU > 50% Remove 3 instances if CPU < 10%
AutoScaling Setup 1. Create an ELB 2. Create AutoScaling launch configuration ID of AMI to be launched Instance type Key pair to authenticate to instances List of EC2 security groups for instances 3. Create an AutoScaling Group 4. Create a trigger for the group
AutoScaling Operation 1. CloudWatch metrics specified in AutoScaling group s trigger are retrieved at specified times. 2. Metrics checked against trigger thresholds If metrics larger than UpperThreshold and the number of instances less than MaxSize, a scaleout event is initiated, launching new instances. If metrics are smaller than LowerThreshold and number of instances is greater than MinSize, a scale-in event is initiated, terminated excess instances.
Key Points 1. Load balancing distributes transactions across multiple servers, CPUs, links, or storage devices 2. Techniques Round robin DNS (configuration) DNS-based GSLB (software + configuration) Layer 4 switch (hardware) Web switch (hardware) Reverse proxy (software) 3. Algorithms Round robin Weighted: least connections/fastest response time Predictive heuristics Random 4. Session stickiness
References 1. Jeff Barr, Host Your Web Site in the Cloud: Amazon Web Services Made Easy, Sitepoint, 2010. 2. Theo Schlossnagle, Scalable Internet Architectures, Sams Publishing, 2007. 3. Willy Tarreau, Making Applications Scalable with Load Balancing, http://1wt.eu/articles/2006_lb/, 2006. 4. Pete Tenereillo, Why DNS Based Global Server Load Balancing (GSLB) Doesn t Work, http://www.tenereillo.com/gslbpageofshame.htm, 2004.