Migrating 200+ Load Balancers Into Mesos. Stephen Salinas

Transcription

1 Migrating 200+ Load Balancers Into Mesos Stephen Salinas

2 Load Balancing

3 Marketing Software Sales Software

4 PaaS At HubSpot Over 150 Engineers Over 2000 deployables ~350 daily deploys to prod

5 api.hubapi.com/ Load Balancer

6 Load Balancer Updates Write Config Files ye_old_deploy.py Start Service Load Balancer

7 Mesos Master api.hubapi.com/ Load Balancer Scheduler

8 Mesos Master / /meetings /reports api.hubapi.com/ /reports / /meetings Scheduler

9 Mesos Master / /meetings /reports api.hubapi.com/ /reports / /meetings Baragon API Load Balancer Updates Singularity Scheduler

10 The Two-Phase Commit host, port, path, domain, extra config Find relevant agents Check for conflicts Result Trigger update on agents Singularity Scheduler Baragon API Respond to Singularity Hold old tasks until they are removed from the LB Don t consider new tasks successfully started unless adding to LB succeeds Response Apply complete request data Write config file from a template Valid? Return success or revert and api.hubapi.com somethingelse.com api.hubapi.com return a failure

11 Mesos Master / /meetings /reports api.hubapi.com/ /reports / /meetings Baragon API Load Balancer Updates Singularity Scheduler

12 Mesos Master / /meetings api.hubapi.com/ ELB /reports /reports / /meetings Baragon API Load Balancer Updates Singularity Scheduler

13 api.hubapi.com/ app.hubspot.com forms.hubspot.com >45 groups signup.hubspot.com nav.hubapi.com >200 separate servers website.grader.com leadin.com library.hubspot.com

14

15

16 Mesos Master / /meetings api.hubapi.com/ ELB /reports /reports / /meetings Baragon API Load Balancer Updates Singularity Scheduler

17 Mesos Master / /meetings /reports api.hubapi.com/ ELB /reports / /meetings Singularity Scheduler Load Balancer Updates Baragon API

18 Why Run LBs in Mesos? Reliability Scalability Easier deployment and upgrades Save $$$

19 Challenges Running LBs in Mesos Dependencies Logging Disposable instances Bootstrapping No manual touch needed Service Discovery Nginx s Nginx ELB

20 Dependencies start.sh Build Nginx + Baragon Agent into a Docker image render basic configs from environment Run a container with just enough information to bootstrap Where is Baragon Service? What group am I in?

21 Logging container task sandbox access.log Log Tailer

22 Service Discovery

23 Service Discovery Each group has a dedicated port Tell the ELB where Nginx is on startup Ensure removal from the ELB before shutting down ELB

24 Startup 1. Phone home with group name 2. Most recent configs for group 3. Agent applies and checks configs Service 4. Ready to serve traffic Agent 5. Add to ELB 6. Traffic starts flowing to new LB ELB 7. Continually ensure ELB in sync with active agents

25 Shutdown 1. KILL or other signal sent to agent 2. Send shutdown notification Service 3. Attempt to remove from ELB Agent 4. start.sh checks for successful removal ELB 5. If not removed, remove health check html file and wait 6. Agent container stops Nginx

26 Problems ELB API rate limiting and ELB inconsistencies Docker + DNS lookups Hard memory limit

27 Benefits Scale with 1 click Automatic replacement of lost LBs by the scheduler Mesos Master < 20 seconds to launch new LB api.hubapi.com/ Condense 200+ EC2 instances to <10, saving ~$24,000/month ~15 man hours saved every Nginx upgrade Singularity Scheduler Load Balancer Updates Baragon API

28

29 Check It Out!