3 What is load balancing? load balancing is a technique to distribute workload evenly across two or more computers, network links, CPUs, hard drives, or other resources, in order to get optimal resource utilization, maximize throughput, minimize response time, and avoid overload. Wikipedia
4 Load Balanced Servers Internet Server 1 Server 2 Server N
5 Load Balancing + Failover
6 Load Balancing + Failover + Cloud
7 Round robin DNS DNS global server load balancing Layer 4 load balancer Web switch Reverse proxy Load balancing techniques
8 Round Robin DNS Configure multiple A records for single name Return next IP address from circular list Some servers may try to return closest IP address. Problems Clients cache addresses. No awareness of availability status of addresses.
9 > dig google.com Round Robin DNS Entry ;; ANSWER SECTION: google.com. 21 IN A google.com. 21 IN A google.com. 21 IN A google.com. 21 IN A google.com. 21 IN A
10 DNS Global Server Load Balancing 1. Client makes a DNS query to ISP DNS server. 2. ISP DNS server looks up DNS records. 3. GSLB receives DNS query. 4. GSLB algorithm examines site health and latency to determine which site best. 5. GSLB returns IP address to client of site that is available and has lowest latency. Problem: DNS servers and clients cache responses for long periods of time.
11 Application Layer 4 Load Balancing Switch Presentation Session Network Address Translation (NAT) for LB One external IP address Switch maps incoming connections to one of the pool of servers. Switch maps outgoing connection source IP address to IP address of the switch. Additional features Better load balancing algorithms Detection of availability status of servers Transport Network Data Link Physical
12 Application Web Switch AKA content switch or application switch Presentation Session Transport Network Data Link Physical Balances load at application layer Multiple load balancing algorithms Handles SSL for servers May also perform compression May also add firewall features
14 Reverse Proxy Proxy server has network identity Forwards requests to backend web servers Features can include caching, compression, SSL
15 Open Source Load Balancers Perlbal used by LiveJournal, TypePad HAProxy used inside hw + sw load balancers Varnish HTTP accelerator, caching + balancing Pound Security-focused HTTP accelerator mod_proxy_balance Apache module NGinx lightweight, high performance web proxy
16 Round robin Least connections/time Predictive Random Weighted strategies Load balancing strategies
17 Round Robin Server 1 Server 5 Server 2 Server 4 Server 3
18 Least Connections / Response Time Send requests to server with the least number of connections or lowest response time. Good for balancing between requests with different requirements or servers with different performance levels. Problems can arise with multiple load balancers making decisions in parallel. Weights can be added for manual tweaking.
19 Predictive Round robin or least connections with additional heuristics to compensate for information staleness issue arising from many short rapid transactions.
20 Random Either select server at random or combine with resource-based algorithm to deal with information staleness problems. Weighted random adds manual constant to probability of choosing particular servers. A dual-core isn t twice as fast as a single CPU. Requires trial and error experimentation.
22 The Problem User interactions with a server have a certain amount of state. Authentication Current stage of transaction Shopping carts, etc. How can the application preserve state when the load balancer sends next request to a different server?
23 Session Stickiness Stickiness requires session-aware load balancer Ensures that future requests from same session always go to same server. Problems Lack of failover since requests tied to single server Difficult to allocate resources effectively since sessions vary tremendously in duration and size
24 Client-side State Why not store all state in client cookies? Insufficient client storage (ameliorated by HTML5) Insecure (can t trust client to control prices, id) Solution Store frequently accessed low security data in client cookies. Store rarely accessed data in central backend storage and take performance hit to access when necessary.
25 Elastic load balancing
27 Elastic Load Balancing Distributes traffic across EC2 instances Instances can be located across multiple AZs Supports any TCP based protocol Monitors instance health and will not distribute traffic to unhealthy instances Supports SSL termination Supports session stickiness Provides metrics to CloudWatch
28 ELB Algorithms 1. Weighted round robin Sends new request to instance handling smallest number of requests. Round robin choice if multiple instances have same smallest number of requests. Can configure health checks to prevent ELB from sending requests to unresponsive instances. 2. Session sticky Weighted round robin, but all requests that are part of a session go to the same server.
29 ELB Lifecycle 1. Create an ELB List of AZs Parameters for health check List of listeners 2. Add instances to ELB By instance ID ELB will track status: InService, OutOfService 3. Advertise public DNS name of ELB 4. Modify number of instances to match traffic Or setup CloudWatch/AutoScale to handle for you 5. Delete ELB when unneeded
30 ELB, CloudWatch, and AutoScale
31 CloudWatch Monitoring service for EC2 CPU utilization, Data transfer, Storage usage Pricing Basic Monitoring with 5 minute granularity free Detailed Monitoring (1 minute) for 1.5 per hour 10 per alarm after first 10 alarms
32 CloudWatch Terminology A namespace represents a source of data AWS/EC2 AWS/ELB A measure is a raw, observed data value One minute s worth of observation A unit is an attribute of a measure Seconds, %, bytes, bits, counts, bytes/s, bits/s A dimension is a refined view of a type of data AvailabilityZone, ImageType, InstanceID, A metric is a stored, processed measure A statistic is a computed attribute of a metric Minimum, maximum, average, sum
33 AutoScaling Group AutoScaling Set of EC2 instances that should scale together Triggers Scale on CloudWatch alerts Scale on time-based schedule Fixed number of healthy instances Examples Add 3 instances if CPU > 50% Remove 3 instances if CPU < 10%
34 AutoScaling Setup 1. Create an ELB 2. Create AutoScaling launch configuration ID of AMI to be launched Instance type Key pair to authenticate to instances List of EC2 security groups for instances 3. Create an AutoScaling Group 4. Create a trigger for the group
35 AutoScaling Operation 1. CloudWatch metrics specified in AutoScaling group s trigger are retrieved at specified times. 2. Metrics checked against trigger thresholds If metrics larger than UpperThreshold and the number of instances less than MaxSize, a scaleout event is initiated, launching new instances. If metrics are smaller than LowerThreshold and number of instances is greater than MinSize, a scale-in event is initiated, terminated excess instances.
36 Key Points 1. Load balancing distributes transactions across multiple servers, CPUs, links, or storage devices 2. Techniques Round robin DNS (configuration) DNS-based GSLB (software + configuration) Layer 4 switch (hardware) Web switch (hardware) Reverse proxy (software) 3. Algorithms Round robin Weighted: least connections/fastest response time Predictive heuristics Random 4. Session stickiness
37 References 1. Jeff Barr, Host Your Web Site in the Cloud: Amazon Web Services Made Easy, Sitepoint, Theo Schlossnagle, Scalable Internet Architectures, Sams Publishing, Willy Tarreau, Making Applications Scalable with Load Balancing, Pete Tenereillo, Why DNS Based Global Server Load Balancing (GSLB) Doesn t Work,
Load Balancing Topics 1. What is load balancing? 2. Load balancing techniques 3. Load balancing strategies 4. Sessions 5. Elastic load balancing What is load balancing? load balancing is a technique to
BASICS OF SCALING: LOAD BALANCERS Lately, I ve been doing a lot of work on systems that require a high degree of scalability to handle large traffic spikes. This has led to a lot of questions from friends
CS 188/219 Scalable Internet Services Andrew Mutz October 8, 2015 For Today About PTEs Empty spots were given out If more spots open up, I will issue more PTEs You must have a group by today. More detail
Monitoring and Scaling My Application In the last chapter, we looked at how we could use Amazon's queuing and notification services to add value to our existing application. We looked at how we could use
Chapter 1 Load Balancing 57 Understanding Slow Start When you configure a NetScaler to use a metric-based LB method such as Least Connections, Least Response Time, Least Bandwidth, Least Packets, or Custom
PES Platform & Engineering Services High Availability Load Balancing in the Agile Infrastructure HEPiX Bologna, April 2013 Vaggelis Atlidakis, -PES/PS Ignacio Reguero, -PES/PS PES Outline Core Concepts
Middleware and Web Services Lecture 11: Cloud Computing Concepts doc. Ing. Tomáš Vitvar, Ph.D. email@example.com @TomasVitvar http://vitvar.com Czech Technical University in Prague Faculty of Information
Web Application Hosting Cloud Architecture Executive Overview This paper describes vendor neutral best practices for hosting web applications using cloud computing. The architectural elements described
FortiBalancer: Global Server Load Balancing WHITE PAPER FORTINET FortiBalancer: Global Server Load Balancing PAGE 2 Introduction Scalability, high availability and performance are critical to the success
Single Pass Load Balancing with Session Persistence in IPv6 Network C. J. (Charlie) Liu Network Operations Charter Communications Load Balancer Today o Load balancing is still in use today. It is now considered
Design for Failure High Availability Architectures using AWS Harish Ganesan Co founder & CTO 8KMiles www.twitter.com/harish11g http://www.linkedin.com/in/harishganesan Sample Use Case Multi tiered LAMP/LAMJ
Department of Information Engineering, CUHK Term 1, 2015/16 IERG 4080 Building Scalable Internet-based Services Lecture 4 Load Balancing Lecturer: Albert C. M. Au Yeung 30 th September, 2015 Web Server
Load Balancer Comparison: a quantitative approach a call for researchers ;) Complex Internet infrastructure high performance systems clusters grids high availability systems resilient storage resilient
White Paper Overview Many enterprises attempt to scale Web and network capacity by deploying additional servers and increased infrastructure at a single location, but centralized architectures are subject
NTT Communications Cloudⁿ Load Balancing Advanced Operation Manual Ver.1.0 Please refrain from any secondary distribution (distributing, copying, or providing this booklet or any similar acts) of the content
Server Traffic Management Jeff Chase Duke University, Department of Computer Science CPS 212: Distributed Information Systems The Server Selection Problem server array A server farm B Which server? Which
White Paper Global Server Load Balancing APV Series Application Delivery Controllers May 2011 Global Server Load Balancing Access. Security. Delivery. Introduction Scalability, high availability and performance
S y s t e m A r c h i t e c t u r e V e r s i o n 5. 0 Page 1 Enterprise etime automates and streamlines the management, collection, and distribution of employee hours, and eliminates the use of manual
Amazon Web Services Primer William Strickland COP 6938 Fall 2012 University of Central Florida AWS Overview Amazon Web Services (AWS) is a collection of varying remote computing provided by Amazon.com.
2 Apache Tomcat Tomcat Clustering: Part 2 Load balancing Mark Thomas, 15 April 2015 Introduction Apache Tomcat committer since December 2003 firstname.lastname@example.org Tomcat 8 release manager Member of the Servlet,
Web Application Hosting in the AWS Cloud Best Practices May 2010 Matt Tavis Page 1 of 12 Abstract Highly-available and scalable web hosting can be a complex and expensive proposition. Traditional scalable
the Availability Digest Redundant Load Balancing for High Availability July 2013 A large data center can comprise hundreds or thousands of servers. These servers must not only be interconnected, but they
CumuLogic Load Balancer Overview Guide March 2013 CumuLogic Load Balancer Overview Guide 1 Table of Contents CumuLogic Load Balancer... 3 Architectural Overview of CumuLogic Load Balancer... 4 How to Use
IOSR Journal of Computer Engineering (IOSR-JCE) e-issn: 2278-0661, p- ISSN: 2278-8727Volume 16, Issue 4, Ver. I (Jul-Aug. 2014), PP 39-46 Auto-Scaling, Load Balancing and Monitoring As service in public
Basics of Cloud Computing Lecture 3 Scaling Applications on the Cloud Satish Srirama Outline Scaling Information Systems Scaling Enterprise Applications in the Cloud Auto Scaling 25/02/2014 Satish Srirama
Web Application Hosting in the AWS Cloud Best Practices September 2012 Matt Tavis, Philip Fitzsimons Page 1 of 14 Abstract Highly available and scalable web hosting can be a complex and expensive proposition.
Getting Started with AWS Web Application Hosting for Amazon Web Services Getting Started with AWS: Web Application Hosting for Amazon Web Services Copyright 2014 Amazon Web Services, Inc. and/or its affiliates.
Apache HTTP Server Load-Balancing with Apache HTTPD 2.2 and later Erik Abele www.eatc.de About Me Working internationally as IT Consultant Areas: Administration & Operations Working on and with Open Source
SiteCelerate white paper Arahe Solutions SITECELERATE OVERVIEW As enterprises increases their investment in Web applications, Portal and websites and as usage of these applications increase, performance
ZEN LOAD BALANCER EE v3.04 DATASHEET The Load Balancing made easy OVERVIEW The global communication and the continuous growth of services provided through the Internet or local infrastructure require to
Content Delivery Networks Silvano Gai Cisco Systems, USA Politecnico di Torino, IT email@example.com 1 Agenda What are Content Delivery Networks? DNS based routing Server Load Balancing Content Routers Ethical
Preparing Your IT for the Holidays A quick start guide to take your e-commerce to the Cloud September 2011 Preparing your IT for the Holidays: Contents Introduction E-Commerce Landscape...2 Introduction
What's New in Cisco ACE Application Control Engine Module for the Cisco Catalyst 6500 and Cisco 7600 Series Software Release 2.1.0 PB458841 Product Overview The Cisco ACE Application Control Engine Module
5 Easy Steps to Implementing Application Load Balancing for Non-Stop Availability and Higher Performance DEPLOYMENT GUIDE Prepared by: Jim Puchbauer Coyote Point Systems Inc. The idea of load balancing
Content Delivery Networks Terena 2000 ftp://ftpeng.cisco.com/sgai/t2000cdn.pdf Silvano Gai Cisco Systems, USA Politecnico di Torino, IT firstname.lastname@example.org Terena 2000 1 Agenda What are Content Delivery Networks?
KeyControl Installation on Amazon Web Services Contents Introduction Deploying an initial KeyControl Server Deploying an Elastic Load Balancer (ELB) Adding a KeyControl node to a cluster in the same availability
Application Delivery Networking. Raj Jain Washington University in Saint Louis Saint Louis, MO 63130 Jain@cse.wustl.edu These slides and audio/video recordings of this class lecture are at: 8-1 Overview
White Paper White Paper June 2013 Resonate Global Dispatch Intelligent Multisite Global Service Level Control Resonate, Inc. 16360 Monterey St., STE 260 Morgan Hill, CA 95037 Phone: 1-408-545-5501 Fax:
Content Switching Module for the Catalyst 6500 and Cisco 7600 Internet Router Product Overview The Cisco Content Switching Module (CSM) is a Catalyst 6500 line card that balances client traffic to farms
High Availability HTTP/S R.P. (Adi) Aditya email@example.com Senior Network Architect HTTP/S is not the Internet HTTP/S Internet so why care about High Availability HTTP/S? because HTTP/S is such a large
WHITE PAPER Enhancing Application Delivery and Load Balancing on Amazon Web Services with Brocade Virtual Traffic Manager ALREADY USING AMAZON ELASTIC LOAD BALANCER? As an abstracted service, Amazon ELB
Industry: Energy; Retail & ecommerce Challenges: Deliver an optimal user experience by balancing traffic across multiple leased links Balance user access and application operation requests among back-end
Implementing Microsoft Office Communications Server 2007 With Coyote Point Systems Equalizer Load Balancing WHITE PAPER Prepared by: Mark Hoffmann Coyote Point Systems Inc. Abstract: This white paper describes
Web Servers Outline Chris Chin, Gregory Seidman, Denise Tso March 19, 2001 I. Introduction A. What is a web server? 1. is it anything that can be retrieved with an URL? 2. (web service architecture diagram)
Développement logiciel pour le Cloud (TLC) 7. Infrastructure-as-a-Service Guillaume Pierre Université de Rennes 1 Fall 2012 http://www.globule.org/~gpierre/ Développement logiciel pour le Cloud (TLC) 1
CS312 Solutions #6 March 13, 2015 Solutions 1. (1pt) Define in detail what a load balancer is and what problem it s trying to solve. Give at least two examples of where using a load balancer might be useful,
ORACLE TRAFFIC DIRECTOR KEY FEATURES AND BENEFITS KEY FEATURES AND BENEFITS FAST, RELIABLE, EASY-TO-USE, SECURE, AND SCALABLE LOAD BALANCER [O.SIDEBAR HEAD] KEY FEATURES Easy to install, configure, and
Data Sheet DPtech ADX Series DPtech ADX Application Delivery Platform Series Overview IT requirements for service capability can be summarized as "acceleration", "security" and "reliability". The contradiction
Basics of Cloud Computing Lecture 3 Scaling Applications on the Cloud Satish Srirama Outline Scaling Information Systems Scaling Enterprise Applications in the Cloud Auto Scaling 3/24/2015 Satish Srirama
The recognized leader in proven and affordable load balancing and application delivery solutions White Paper 7 Easy Steps to Implementing Application Load Balancing For 100% Availability and Accelerated
Accelerating Web-Based SQL Server Applications with SafePeak Plug and Play Dynamic Database Caching A SafePeak Whitepaper February 2014 www.safepeak.com Copyright. SafePeak Technologies 2014 Contents Objective...
Oracle Collaboration Suite Firewall and Load Balancer Architecture Release 2 (9.0.4) Part No. B15609-01 November 2004 This document discusses the use of firewall and load balancer components with Oracle
Chapter 11 Cloud Application Development Contents Motivation. Connecting clients to instances through firewalls. Chapter 10 2 Motivation Some of the questions of interest to application developers: How
Practical Load Balancing Ride the Performance Tiger Illtil Peter Membrey David Hows Eelco Plugge Apress8 Contents About the Authors About the Technical Reviewers Special Thanks to serverlove Acknowledgments
Ranch Networks for Hosted Data Centers Internet Zone RN20 Server Farm DNS Zone DNS Server Farm FTP Zone FTP Server Farm Customer 1 Customer 2 L2 Switch Customer 3 Customer 4 Customer 5 Customer 6 Ranch
Connection Setup Process makes a REST call using the user s access key to chooses a repeater for establishing a secure connection for Local Testing. The repeater exists within the BrowserStack cloud infrastructure.
P3 InfoTech Solutions Pvt. Ltd http://www.p3infotech.in July 2013 Created by P3 InfoTech Solutions Pvt. Ltd., http://p3infotech.in 1 Web Application Deployment in the Cloud Using Amazon Web Services From
Web Caching and CDNs Aditya Akella 1 Where can bottlenecks occur? First mile: client to its ISPs Last mile: server to its ISP Server: compute/memory limitations ISP interconnections/peerings: congestion
Creating Web Farms with Linux (Linux High Availability and Scalability) Horms (Simon Horman) firstname.lastname@example.org December 2001 For Presentation in Tokyo, Japan http://verge.net.au/linux/has/ http://ultramonkey.org/
Lasse Pettersson LoadBalancer and Exchange 2013 Lasse Pettersson Load Balancing Load Balancing basics Load balance previous version of Exchange Load Balance Exchange 2013 introduction What is LoadBalancing?
MEASURING WORKLOAD PERFORMANCE IS THE INFRASTRUCTURE A PROBLEM? Ashutosh Shinde Performance Architect email@example.com Validating if the workload generated by the load generating tools is applied
ArcGIS 10.3 Server on Amazon Web Services Copyright 1995-2015 Esri. All rights reserved. Table of Contents Introduction What is ArcGIS Server on Amazon Web Services?............................... 5 Quick
Scalability of web applications CSCI 470: Web Science Keith Vertanen Scalability questions Overview What's important in order to build scalable web sites? High availability vs. load balancing Approaches
Amazon EC2 Product Details Page 1 of 5 Amazon EC2 Functionality Amazon EC2 presents a true virtual computing environment, allowing you to use web service interfaces to launch instances with a variety of
VERITAS Cluster Server Traffic Director Option Product Overview V E R I T A S W H I T E P A P E R Table of Contents Traffic Director Option for VERITAS Cluster Server Overview.............................................1
1. Comments on reviews a. Need to avoid just summarizing web page asks you for: i. A one or two sentence summary of the paper ii. A description of the problem they were trying to solve iii. A summary of
ZEN LOAD BALANCER EE v3.02 DATASHEET The Load Balancing made easy OVERVIEW The global communication and the continuous growth of services provided through the Internet or local infrastructure require to
Identikey Server Performance and Deployment Guide 3.1 Disclaimer of Warranties and Limitations of Liabilities Disclaimer of Warranties and Limitations of Liabilities The Product is provided on an 'as is'
15-319 / 15-619 Cloud Computing Recitation 5 September 29 th & October 1 st 2015 1 Overview Administrative Stuff Last Week s Reflection Project 2.1, OLI Modules 5 & 6, Quiz 3 New concepts This week s schedule
Scalable Web Applications Reference Architectures and Best Practices Brian Adler, PS Architect 1 Scalable Web Application 2 1 Scalable Web Application What? An application built on an architecture that
SonicWALL NAT Load Balancing Overview This feature module will detail how to configure the Network Address Translation (NAT) & Load Balancing (LB) features in SonicOS Enhanced 4.0 and newer, to balance