Redundancy Doesn't Always Mean "HA" or "Cluster"

Transcription

1 Redundancy Doesn't Always Mean "HA" or "Cluster" A cautionary tale against using hammers to solve all redundancy and resiliency problems... OpenStack Design Summit Oct 2012 Randy CTO, Cloudscaling Dan Sr. Engineer, Cloudscaling CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution* * All unlicensed or borrowed works retain their original licenses 1

2 Our Journey Today 1. HA pairs are not the only type of redundancy 2. Alternative redundancy patterns for HA 3. Redundancy patterns in Open Cloud System* * Cloudscaling s OpenStack-powered cloud operating system ( distribution ) 2

3 What Do We Mean By HA? We mean what most people mean... Two servers or network devices that look like one 3

4 HA HA? HA pairs come in a couple flavors Active / Passive 4

5 HA HA? People like this flavor best, but it s not always possible... Active / Active 5

6 HA HA HA HA HA?? Many people wish they could get it more like this... HA cluster aka massive operational nightmare 6

7 Cluster<bleep>! Imagine this was 4 or 6 nodes in the cluster 4 network tech. 7 NICs / node A million different ways to break 7

8 HA Pairs Are One Type of Redundancy Herein lies the problem... 8

9 The Problem With HA -mmers There are many, but these two matter most... Catastrophic failures No scale out 9

10 HA Pairs Have Binary Failures Either working or dead, nothing in-between 10

11 What is Scale-out? A B A B C D N A B Scale-up - Make boxes bigger (usually an HA pair) Scale-out - Make moar boxes 11

12 Scaling out is a mindset Scaling up is like treating your servers as pets bowzer.company.com web001.company.com Servers *are* cattle 12

13 HA Pair Failures* - 100% down Hardware rarely fails, operators fail, software fails Who Type Year Why Duration Apple Switch 2005 Bug 2 hrs Flexiscale SAN 2007 Ops Err 24 hrs Vendio NAS 2008 Ops Err 8 hrs UOL Brazil SAN 2011 Bug 72 hrs Twitter Datacenter 2012 Bug+Ops 2 hrs * This is a handful of examples as a baseline; I m sure you can find many more 13

14 HA Pairs Are an All-in Move They better not fail... 14

15 Risk Reduction Many small failure domains is usually better 15

16 Big failure domains vs. small Would you rather have the whole cloud down or just a small bit for a short period of time? Still a scale-up pattern... wouldn t you rather scale-out? 16

17 Pair vs. Scale-out Load Balancing No scale-out State Sync (100% loss) Shared-nothing Architecture (20% loss) 17

18 Pair vs. Scale-out Load Balancing No scale-out State Sync (100% loss) Shared-nothing Architecture (20% loss) 17

19 What s Usually an HA Pair in OpenStack? Everything... Service Endpoints (APIs) Messaging System (RPC) Worker Threads (e.g. Scheduler, Networking) Database (MySQL) 18

20 What needs to be an HA pair? Not much needs state synchronization Service Endpoints (APIs) Messaging System (RPC) Worker Threads (e.g. Scheduler, Networking) Database (MySQL) 19

21 Fault Tolerance Methodologies 20

22 Fault Tolerance in OCS 21

23 Service Distribution High Availability Without Compromise Resilient Stateless Scale-out 22

24 Service Distribution Combines Standard Networking Technologies OSPF /etc/quagga/ospfd.conf router ospf ospf router-id network area Anycast /etc/quagga/zebra.conf interface lo:2 description Pound listening address ip address /32 Load- Balancing Proxy /etc/pound/pound.conf ListenHTTP Address Port 8774 xhttp 1 Service BackEnd Address Port 8774 End BackEnd Address Port 8774 End End End 23

25 Resilient OpenStack Horizontally Scalable, No Single Point Of Failure Service Distribution ZeroMQ Service Endpoints (APIs) Messaging System (RPC) Service Distribution Worker Threads (e.g. Scheduler, Networking) MMR + HA Database (MySQL)

26 Service Distribution Advantages What Makes This a Superior Solution? True horizontal scalability with no centralized controller Services are always running, failover is nearly instant Reduced complexity, fewer idle resources No need for separate load balancers Server Server Server Server Server Server Server... Failover vs. Distributed Services 25

27 Perfect For Site Resiliency Service Distribution Works With Multiple Sites Traditional HA pairs do not support cross-site resiliency Service Distribution fail across sites without DNS redirections 26

28 Service Distribution in Action Example: Distributed Load Balancing 1) OSPF OSPF Router(s) OSPF advertisement OSPF advertisement V Quagga Quagga HTTP Proxy HTTP Proxy 27

29 Service Distribution in Action Example: Distributed Load Balancing 1) OSPF OSPF Router(s) 2) ECMP Per-flow Load Balancing 3) Load-balancing HTTP Proxy OSPF advertisement V Quagga Per-Flow Load Balancing OSPF advertisement Quagga HTTP Proxy HTTP Proxy 28

30 Service Distribution in Action Example: Distributed Load Balancing 1) OSPF OSPF Router(s) 2) ECMP Per-flow Load Balancing 3) Load-balancing HTTP Proxy OSPF advertisement V Quagga Per-Flow Load Balancing OSPF advertisement Quagga HTTP Proxy HTTP Proxy 4) Unlimited # of Back-End Servers Server Server Server Server 29

31 Failure Resiliency Client Client Client Client Load Balancer/ Proxy Proxy Load Balancer/ Proxy Load Balancer/ Proxy Load Balancer/ Proxy Server Server Server Server Server Server Server Server Server Server 10% Load Each Server 30

32 Failure Resiliency Client Client Client Client X Load Balancer/ Proxy Proxy Load Balancer/ Load Balancer/ Proxy Load Balancer/ Proxy Server Server Server Server Load Balancer/ Proxy Server Server Server Server Server Server 10% Load Each Server 31

33 Failure Resiliency Client Client Client Client Load Balancer/ Proxy Proxy Load Balancer/ Proxy Load Balancer/ Proxy Load Balancer/ Proxy X Server Server Server Server Server Server Server Server Server Server 10% Increased Server Load 32

34 OCS NAT Service Example: Scale-out Network Address Translation BGP Multiple ISP providers NAT Service Distribution VMs 33

35 Brokerless Messaging With ZeroMQ Avoiding RabbitMQ s Single Point Of Failure Nova-Compute Single Point Of Failure RabbitMQ Broker Nova-Scheduler RabbitMQ (Brokered) Nova-API 34

36 Brokerless Messaging With ZeroMQ Avoiding RabbitMQ s Single Point Of Failure Nova-Compute Nova-Compute Single Point Of Failure RabbitMQ Broker Nova-Scheduler RabbitMQ (Brokered) Nova-API Nova-Scheduler vs. Nova-API ZeroMQ (Peer To Peer) 35

37 What did we learn today? 1. HA-mmers are for nails 2. Scale-out rules for redundancy 3. Design-for-failure is a mentality, not a pair 4. Resiliency over redundancy 36

38 Q&A Randy CTO, Cloudscaling Dan Sr. Engineer, Cloudscaling OCS 2.0 Public Cloud Benefits Private Cloud Control Open Cloud Economics 37