Building the Drupal Borg

Transcription

1 Building the Drupal Borg BADCamp October 23nd 2011

2 In The Beginning Scaling used to be far more complex.

3 The Late 90 s Ease of impleme ntation Java v1 Use-Cases Covered

4 The Late 90 s Ease of impleme ntation J2EE Java v1 Use-Cases Covered

5 The Late 90 s Ease of impleme ntation J2EE Java v1 Microsoft Stack Use-Cases Covered

6 The Late 90 s LAMP Ease of impleme ntation J2EE Java v1 Microsoft Stack Use-Cases Covered

7 A Dark Time All apps written from scratch Network databases still immature Publishing HTML still the goal Remember the web in the 90 s? (sorta sucked)

8 LAMP Shines a Light Share-nothing runtime Network-connected database Free and Open Source Runs on any hardware

9 And so we grew And so the web grew

10 Web Scale C10k Challenge Evented I/O wins Nginx and Lighttpd Memcache Distributed reads

11 But the Architecture

12 By net. LiveJournal Backend: Today Roughly. BIG-IP bigip1 bigip2 perlbal (httpd/proxy) proxy1 mod_perl Global Database master_a master_b proxy2 web1 proxy3 proxy4 web2 web3 Memcached mc1 slave1 slave2... slave5 Mogile Storage Nodes sto1 sto2... sto8 proxy5 web4... web50 mc2 mc3 mc4... mc12 User DB Cluster 1 uc1a uc1b User DB Cluster 2 uc2a uc2b User DB Cluster 3 uc3a uc3b Mogile Trackers tracker1 tracker2 User DB Cluster 4 uc4a uc4b MogileFS Database mog_a mog_b User DB Cluster 5 uc5a uc5b

13 Infrastructure Ease of Shared Hosting impleme VPS ntation Dedicated Server HA Web Cluster Custom Cluster Use-Cases Covered

14 Infrastructure Cost is a real Barrier Ease of Shared Hosting impleme VPS ntation Dedicated Server HA Web Cluster Custom Cluster Use-Cases Covered

15 Infrastructure Expertise is a real barrier Ease of Shared Hosting impleme VPS ntation Dedicated Server HA Web Cluster Custom Cluster Use-Cases Covered

16 The Units of Scale Entire hardware servers is the wrong unit to scale. Tons of capacity at a chunk.

17 The Challenge This isn t just about scaling one website

18 The Challenge This is about scaling the web

19 The Challenge Limited developer talent limits Drupal adoption

20 The Challenge Expertise in infrastructure is even more limited

21 The Challenge Luckily infrastructure is friendly to automation

22 Bring on the Borg!

23 Bring on the Borg!

24 A Brief History of Borg Google / AppEngine CfEngine, Puppet, Chef Heroku

25 This is how we do it

26 This is how we do it

27 Pantheon is the best of both worlds

28 Pantheon will un-block Drupal adoption on infrastructure

29 Inside Pantheon The High Level View

30 Routing Mesh Varnish Firewall Styx Request

31 Routing Mesh Routing Mesh

32 Routing Mesh Yggdrasil DNS Routing Mesh

33 Routing Mesh Yggdrasil DNS Routing Mesh Cassandra

34 Routing Mesh Yggdrasil DNS Routing Mesh Cassandra

35 Routing Mesh Yggdrasil DNS Routing Mesh Cassandra

36 Routing Mesh Routing Mesh Yggdrasil

37 Routing Mesh Routing Mesh Yggdrasil DROPs Matrix

38 DROPs Provision, Isolate and Scale at the per-process level.

39 DROPs Provision, Isolate and Scale at the per-process level.

40 DROPs We can scale this too

41 DROPs We can scale this too

42 DROPs We can scale this too

43 Anatomy of a DROP Nginx PHP-FPM APC Cache Service Bindings

44 Valhalla Dev Test Live Twisted DAV

45 Valhalla Dev Test Live Twisted DAV Cassandra

46 Other Service Bindings MySQL MemCache Solr Redis? Cassandra?

47 Drush Access Custom SSH Server cc all

48 Drush Access Yggdrasil DNS Custom SSH Server cc all

49 Magic On The Horizon

50 Have questions? Contact Zack Rosen