CONSUL AS A MONITORING SERVICE

Transcription

1 CONSUL AS A MONITORING SERVICE

2 SETH

3

4 SERVICE ORIENTED ARCHITECTURE

5 SOA PRIMER Autonomous Limited Scope Loose Coupling

6 ORDER PROCESSING ORDER WEB APP HISTORY FORECASTING

7 ORDER PROCESSING WEB APP DISCOVERY Which nodes are part of "order processing"?

8 ORDER PROCESSING NODE 1 WEB APP NODE 2 NODE N LOAD BALANCING How to ensure request leveling across providers?

9 ORDER PROCESSING NODE 1 WEB APP NODE 2 LOAD BALANCER NODE N ANTI- PATTERN Load Balancer is a Single Point of Failure (SPOF)

10 ORDER PROCESSING NODE 1 WEB APP NODE 2 LOAD BALANCER NODE 3 HEALTH CHECKING How to avoid routing to unhealthy hosts?

11 maintenance: false feature_a: true role: "web" WEB 1 WEB APP WEB 2 WEB N CONFIGURATION How to efficiently push dynamic configuration?

12 4 BASIC PROBLEMS SERVICE LOAD HEALTH KEY- VALUE DISCOVERY BALANCING CHECKING CONFIGURATION

13 EXISTING "SOLUTIONS" ZOOKEEPER ETCD SENSU SMART STACK

14 CONSUL

15 Service Discovery HTTP + DNS

16 demo master dig web-frontend.service.consul

17 demo master dig web-frontend.service.consul ; <<>> DiG P1 <<>> web-frontend.service.consul. ANY ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;web-frontend.service.consul. IN ANY ;; ANSWER SECTION: web-frontend.service.consul. 0 IN A web-frontend.service.consul. 0 IN A

18 Datacenter Aware

19 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT RPC LAN GOSSIP RPC SERVER SERVER SERVER REPLICATION REPLICATION

20 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT RPC LAN GOSSIP RPC SERVER SERVER SERVER REPLICATION REPLICATION SERVER REPLICATION SERVER REPLICATION SERVER

21 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT RPC LAN GOSSIP RPC SERVER SERVER SERVER REPLICATION REPLICATION WAN GOSSIP SERVER REPLICATION SERVER REPLICATION SERVER

22 Host & Service Level Health Checks

23 demo master consul-template -template="example.ctmpl" -dry > listen http-in bind *:8000 server web :80 server web :80 server web :80 demo master

24 demo master consul-template -template="example.ctmpl" -dry > listen http-in bind *:8000 server web :80 server web :80 server web :80 demo master sudo stop webserver

25 demo master consul-template -template="example.ctmpl" -dry > listen http-in bind *:8000 server web :80 server web :80 demo master sudo stop webserver

26 demo master consul-template -template="example.ctmpl" -dry > listen http-in bind *:8000 server web :80 server web :80 server web :80 demo master sudo start webserver

27 K/V Store HTTP API

28 demo master curl -X PUT -d 'bar' true

29 demo master curl -X PUT -d 'bar' true demo master curl [ ] { } "CreateIndex": 100, "ModifyIndex": 200, "Key": "foo", "Flags": 0, "Value": "YmFy"

30 TRUSTED BY

31 HEALTH CHECKS

32 WHAT IS A CHECK? Any command that returns an exit code

33 WHAT IS A CHECK? Any command that returns an exit code 0 1 PASSING WARNING FAILING

34 WHAT IS A CHECK? Output is captured as a "note" for inspection $ curl curl: (7) Failed to connect to port 4455: Connection refused

35 CREATING A CHECK Use a custom script { } "check": { "id": "mem-util", "name": "Memory utilization", "script": "/usr/local/bin/check_mem.py", "interval": "10s" }

36 CREATING A CHECK Use a built-in check type { } "check": { "id": "api", "name": "HTTP API on port 4455", "http": " "interval": "10s", "timeout": "1s" }

37 RESPONSIVE

38

39

40 WEB 1 MONITORING WEB 2 SERVICE WEB N TRADITIONAL MONITORING Pushes information into a silo

41 WEB 1 MONITORING WEB 2 SERVICE WEB N TRADITIONAL MONITORING Pushes information into a silo

42 WEB 1 MONITORING SERVICE WEB 2 WEB N TRADITIONAL MONITORING Pushes information into a silo

43 WEB 1 MONITORING SERVICE WEB 2 WEB N TRADITIONAL MONITORING Pushes information into a silo

44 U WEB 1 MONITORING SERVICE WEB 2 WEB N TRADITIONAL MONITORING Pushes information into a silo

45 U F WEB 1 MONITORING SERVICE WEB 2 WEB N TRADITIONAL MONITORING Pushes information into a silo

46 U F WEB 1 MONITORING SERVICE WEB 2 WEB N TRADITIONAL MONITORING Pushes information into a silo

47 WEB 1 CONSUL WEB 2 WEB N CONSUL MONITORING Removes unhealthy nodes from service discovery layer

48 WEB 1 CONSUL WEB 2 WEB N CONSUL MONITORING Removes unhealthy nodes from service discovery layer

49 dig web.service.consul WEB 1 CONSUL WEB 2 WEB N CONSUL MONITORING Removes unhealthy nodes from service discovery layer

50 dig web.service.consul WEB 1 CONSUL WEB 2 WEB N CONSUL MONITORING Removes unhealthy nodes from service discovery layer

51 dig web.service.consul WEB 1 CONSUL WEB 2 WEB N CONSUL MONITORING Removes unhealthy nodes from service discovery layer

52 host: web.service.consul dig web.service.consul WEB 1 CONSUL WEB 2 WEB N CONSUL MONITORING Removes unhealthy nodes from service discovery layer

53 host: web.service.consul dig web.service.consul WEB 1 CONSUL WEB 2 WEB N CONSUL MONITORING Removes unhealthy nodes from service discovery layer

54 host: web.service.consul dig web.service.consul WEB 1 CONSUL WEB 2 WEB N CONSUL MONITORING Removes unhealthy nodes from service discovery layer

55 host: web.service.consul dig web.service.consul WEB 1 CONSUL WEB 2 WEB N CONSUL MONITORING Removes unhealthy nodes from service discovery layer

56 host: web.service.consul dig web.service.consul WEB 1 CONSUL WEB 2 WEB N CONSUL MONITORING Removes unhealthy nodes from service discovery layer

57 CONSUL MONITORING Unhealthy nodes are not returned from DNS queries dig web.service.consul web-01, web-02, web-03

58 CONSUL MONITORING Unhealthy nodes are not returned from HTTP API curl /v1/services/web web-01, web-02, web-03

59 LOCKING

60 CONSUL LOCK Allows for a new kind of "HA" demo master consul lock [options] prefix child...

61 VAULT 1 VAULT 2 CONSUL VAULT 3 CONSUL LOCK Making standby HA much simpler

62 VAULT 1 L VAULT 2 CONSUL VAULT 3 L CONSUL LOCK Making standby HA much simpler

63 VAULT 1 L VAULT 2 CONSUL VAULT 3 CONSUL LOCK Making standby HA much simpler

64 VAULT 1 L LEADER ELECTION VAULT 2 CONSUL VAULT 3 CONSUL LOCK Making standby HA much simpler

65 VAULT 1 L VAULT 2 CONSUL GET /secret/foo VAULT 3 REQUEST CONSUL LOCK Making standby HA much simpler

66 VAULT 1 L VAULT 2 CONSUL GET /secret/foo VAULT 3 REQUEST CONSUL LOCK Making standby HA much simpler

67 VAULT 1 L VAULT 2 CONSUL GET /secret/foo VAULT 3 REQUEST CONSUL LOCK Making standby HA much simpler

68 VAULT 1 L VAULT 2 CONSUL VAULT 3 CONSUL LOCK Making standby HA much simpler

69 VAULT 1 l VAULT 2 CONSUL VAULT 3 CONSUL LOCK Making standby HA much simpler

70 VAULT 1 VAULT 2 CONSUL VAULT 3 L CONSUL LOCK Making standby HA much simpler

71 VAULT 1 VAULT 2 CONSUL VAULT 3 L CONSUL LOCK Making standby HA much simpler

72 VAULT 1 VAULT 2 CONSUL GET /secret/foo VAULT 3 L REQUEST CONSUL LOCK Making standby HA much simpler

73 VAULT 1 VAULT 2 CONSUL GET /secret/foo VAULT 3 L REQUEST CONSUL LOCK Making standby HA much simpler

74 CONSUL LOCK Solves the "exactly one of these must always be running" problem

75 CONSUL LOCK Also great as a semaphore - rolling restarts

76 SCALABILITY

77 WEB 1 "I'm healthy" MONITORING WEB 2 SERVICE "Good, thanks for asking!" WEB N TRADITIONAL MONITORING Notifies/polls all statuses

78 WEB 1 MONITORING SERVICE 1,000'S OF REQUESTS WEB 2 WEB 1,000 TRADITIONAL MONITORING Notifies/polls all statuses

79 HA WEB 1 MONITORING SERVICE 1,000'S OF REQUESTS WEB 2 WEB 1,000 TRADITIONAL MONITORING Notifies/polls all statuses

80 My status has changed WEB 1 CONSUL WEB 2 WEB N CONSUL MONITORING Notifies on status changes

81 WEB 1 CONSUL 10'S OF REQUESTS WEB 2 WEB 1,000 CONSUL MONITORING Notifies on status changes

82 CONCLUSION

83 SOLVES 4 BASIC PROBLEMS SERVICE LOAD HEALTH KEY- VALUE DISCOVERY BALANCING CHECKING CONFIGURATION

84 SOLVES 4 MORE PROBLEMS G L H 9 LEADER ELECTION SEMAPHORE LOCKING RESPONSIVE SCALABLE

85 SETH QUESTIONS?