Deploying and Monitoring Ruby on Rails

Transcription

1 Mit hotfixes von Carsten Bormann Deploying and Monitoring Ruby on Rails A practical guide Mathias Meyer and Jonathan Weiss, Peritor GmbH Danke!

2 Mashed up with Deployment/Scaling/ Caching/Debugging jan krutisch mindmatters gmbh&co. kg

3 Deployment

4 Real artists ship!

5 Komponenten

6 client (Browser) rails database

7 client (Browser) Heute: rails thin database mongrel ( unicorn)

8 mongrel ist schnell

9 apache ist schneller

10 client webserver rails Dateisystem public/ database

11 rails ist nicht threadsafe

12 client rails webserver rails database Dateisystem public/ rails

13 webserver rails

14 webserver rails

15 oldschool: CGI / SCGI / FastCGI

16 newschool: http

17 mongrel versus...

18 mod_proxy_balancer (in Apache/nginx/lighttpd)

19 oder ein richtiger proxy

20 oder ein richtiger balancer

21 Varnish HAProxy, Squid, etc.

22 true school: passenger

23 Apache-Modul

24 client rails webserver rails database rails Dateisystem public/ mod_passenger

25 automagisch. it just works.

26

27 Infrastructure Infrastructure 5

28 Simple Rails Setup One Rails instance handles all requests Rails is single-threaded: There is only one concurrent request 6

29 Rails Setup 7

30 Rails Setup 8

31 Typical Rails Setup! A load-balancer distributes the incoming requests! Some load-balancers will deliver static requests themselves! Several Rails instances handle all requests! Number of concurrent requests equals number of Rails instances 9

32 Rails Setup Options 10

33 Deployment Questions Mongrel? mod_proxy_balancer? Proxy? Apache? Pound? FastCGI? Load-balancer? Nginx? mod_rails? HA-Proxy? Ebb? Reverse Proxy? Swiftiply? Phusion Passenger? Thin? Rails Application Server? Lighttpd? Pen? 11

34 What we are going to cover today Rails Application Server! FastCGI! Mongrel! mod_rails / Phussion Passenger! JRuby + Glassfish & Co. Proxy/Web Server! Apache2! Nginx! Lighttpd! HA-Proxy 12

35 FastCGI 13

36 FastCGI! Protocol to communicate with long-running CGI applications! Usage of either mod_fcgi with Apache 1.3 or mod_fcgi with Lighttpd! Proxy local and remote FastCGI instances! Oldest way of deploying Rails! Deprecated and unstable! Hard to debug (FastCGI protocol) 14

37 FastCGI 15

38 FastCGI 16

39 17

40 Mongrel 18

41 Mongrel! Developed by Zed Shaw as an alternative to FastCGI! Complete HTTP-Server that can load arbitrary Ruby-servlets! Built-in Rails support 19

42 Mongrel 20

43 Mongrel Apache 2.2 mod_proxy_balancer Lighttpd Nginx HA-Proxy Pound 21

44 Mongrel Cluster Utility to manage several Mongrel instances 22

45 Mongrel Cluster Control Mongrels 23

46 Mongrel and Apache 2.2 Define Mongrel Cluster in Apache 24

47 Simple Mongrel and Apache 2.2 Redirect all traffic to the Mongel cluster 25

48 A more complex example! Redirect dynamic requests! Serve static content! Support cached pages! Support maintenance page! Enable client-side caching of images, stylesheets, and JavaScript! Compress output if supported 26

49 Mongrel! Very robust! Strict HTTP parser! Easy to debug (HTTP!)! Used to be defacto deployment setup with Apache 2.2 and mod_proxy_balancer! Can be a bit difficult to setup (mongrel_cluster, ports, Apache)! Not so easy on mass/virtual hosting 27

50 mod_rails 28

51 mod_rails a.k.a Phusion Passenger! Module for Apache 2.2 (and Nginx)! Allows Apache to control Rails instances! Apache starts and stops application instances depending on the application load! Able to run any Rack-compatible Ruby application (Merb, Sinatra & Co.)! Only manages Rails on one host - no remote instances! Combine with HTTP-Proxy / balancing solution 29

52 Install Phusion Passenger Install Apache module Load and activate in Apache 30

53 Install Phusion Passenger Install Passenger with nginx (installs custom nginx) Configure nginx 31

54 Customized Phusion Passenger Control Rails instance number 32

55 Control Phusion Passenger Restart after deployment: 33

56 Phusion Passenger One machine 34

57 Phusion Passenger One machine Multiple machines 35

58 Phusion Passenger! Ready for production! Makes setup easier on the single machine level! Multiple servers still require load balancer! Suitable for mass-hosting! upcoming standard way of deploying Rails 36

59 JRuby 37

60 ! Ruby Runtime on the Java Virtual Machine! Implemented in Java and Ruby! Compiles Ruby into Java-bytecode! Integrates with Java code and libraries! Java s promises of native threads and JIT! Allows for Ruby/Rails applications to be packaged as WAR files! WAR files deployable on any J2EE-container: Glassfish, JBoss, Tomcat, Jetty, 38

61 JRuby on Rails 39

62 JRuby on Glassfish One machine 40

63 JRuby on Glassfish One machine Multiple machines 41

64 Setup JRuby on Glassfish 1.! Download JRuby and Glassfish 2.! From 42

65 Warble Configuration Define min/max Rails runtimes 43

66 Glassfish for development Shrunk-down version of Glassfish v3 - packaged as a gem Can run any Rack-based application Installation:! $ jruby -S gem install glassfish! $ glassfish! There is no Step 3 44

67 Rails Setup 45

68 Proxy Options 46

69 Proxy Requirements! Hide cluster backend from the user! Load-balancer backend instances! Recognize down hosts! Fair scheduler! (Deliver static content) 47

70 Apache 2.2! Apache 2.2 introduced mod_proxy_balancer! mod_proxy_balancer can speak to multiple backends and balance requests! Apache can acts as a pure proxy or can also serve static files 48

73 Apache 2.2 Pro! Stable, robust, and mature! Many people know how to work with Apache! Integrates well with other modules (SVN, DAV, Auth, ) Con! Apache can be complicated to configure! The stock Apache is quite resource-hungry compared to pure proxy solutions 51

74 Nginx From Russia with love! Nginx - popular Russian webserver with good proxy support! Can load-balance multiple backends and deliver static content! Quite popular with Mongrel as the Rails backend! Recent Passenger support 52

75 Nginx Configuration Simple proxy example Get complete version here: 53

76 Nginx Pro! Stable, robust, and fast! Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files! Simpler configuration file! Can directly talk to memcached - SSI Con! More documentation would be nice! No equivalent for many Apache modules 54

77 Lighttpd! Lightweight and fast webserver! Balancing proxy support! Good FastCGI support! Used to be popular until Mongrel came around 55

78 Lighttpd Configuration Simple proxy example 56

79 Lighttpd Pro! Fast and lightweight! Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files! Simpler configuration file Con! Unstable for some people! Slow development cycle! More documentation would be nice! Configuration file can be too simple (virtual host aliasing)! No equivalent for many Apache modules! (No Passenger support) 57

80 HA-Proxy! HAProxy reliable, high performance TCP/HTTP load balancer! Proxying and content inspection! No content serving, just a proxy! Mature proxy module (fair scheduler)! ACL support See also similar Pound and Pen 58

81 HAProxy Simple proxy example 59

82 HAProxy Pro! Mature, stable, robust, and fast! TCP and HTTP balancing Con! Few Rails examples! Usually not needed in a Rails setup 60

83 Recommended Setups 61

84 Small Site Recommendation Apache 2.2 with mod_rails / Phusion Passenger 62

85 Medium Site Recommendation! Apache/Nginx as the frontend proxy! Passenger/mod_rails as the backend! Deliver static files with Apache 63

86 Large Rails Setup Recommendation! Redundant load-balancer! Redundant proxy / web! Passenger/mod_rails 64

87 Heavy Static Files Recommendation! Deliver static assets through separate web server farm! Passenger/mod_rails 65

88 Java Shop Recommendation! Deliver a Rails-WAR file and you are done! Integrate with existing Java landscape and infrastructure 66

89 67

90 Application Server Handler 68

93 Rack 71

94 Rack Handler 72

95 Rack 73

96 Rack Middleware 74

97 Rack Middleware! Authentication (HTTP, OpenID)! Sessions! Routing! Caching! Logging! Debugging 75

98 Support Frameworks! Rails! Merb! Camping! Sinatra! Mack! Maveric! Halcyon! Ramaze! Rum! Vintage! Waves Web Server! CGI! FastCGI! Mongrel! Phusion Passenger! JRuby Servlets! WEBrick! Ebb! Thin! LiteSpeed! Swiftcore! Unicorn 76

99 Remarks 77

100 Ruby Enterprise Edition! Copy-On-Write patches to Ruby 1.8! Saves memory when spawning several Rails instances! Used by Phusion Passenger if available Dead:

101 Thin, Ebb, Evented Mongrel & Co.! Alternatives to Mongrel! Claim to be faster, lighter, and what have you! Rendering Hello World is usually not your bottleneck Stick with stable and robust Mongrel, or better yet, with Passenger Might be somewhat dated

102 deploy-prozess

103 prozesse (mongrels) müssen neugestartet werden.

104 Bei Passenger: touch tmp/restart.txt

105 capistrano

106 > cap deploy:migrations

107 konfiguration in rubydateien. Yay!

108 set :application, "name" set :repository, " role :app, "mog.i.net" role :web, "mog.i.net" role :db, "mog.i.net", :primary => true require 'lib/cap/backgroundrb_recipes' after 'deploy:update_code', :set_symlinks task :set_symlinks do run "ln -s #{shared_path}/config/database.yml #{release_path}/config/ database.yml" run "ln -s #{shared_path}/data #{release_path}/public/data" end

109 AWE13 Gruppe/04 danke!

110 AWE13 Gruppe/04 danke!

111

112 Erweitern um eigene "Recipes"

113 Kommandos werden parallel auf allen Servern ausgeführt

114 per SSH

115 Unterstützung diverser SCM-Systeme

116 Erlaubt (relativ) einfache rollbacks, jeder deploystand wird abgelegt.

117 Quasi-Standard

118 Nicht auf Rails beschränkt!

119 Infrastructure Deployment 80

120 Deployment Options 82

124 What does Capistrano do? 86

125 Capistrano Deployment Cycle 87

126 Requirements 88

127 What doesn t Capistrano do?! Plan your initial server setup! Configure basic services 89

128 Basic Ingredients! The cap command! Variables! Roles! Tasks! Namespaces 90

129 Basic Ingredients - cap Your one-stop deployment shop 91

130 Basic Ingredients - Variables! Configure basic project information! Override Capistrano s default assumptions! Once set, variables are available globally! Defined using the set method 92

131 Basic Ingredients - Roles! Define types of servers! Default roles! :www! :app! :db! All can point to the same server! But all three must be defined! At least one database server needs to be primary 93

132 Basic Ingredients - Roles Define custom roles as you please Can be reused when defining tasks 94

133 Basic Ingredients - Tasks! Define an atomic set of actions for Capistrano! Can be called from the command line! Or other tasks 95

134 Basic Ingredients - Tasks To find all the tasks available in your project, use 96

135 Basic Ingredients - Namespaces Group tasks together logically Namespaces and tasks are separated with : 97

136 Get Your Capistrano On 98

137 Get Your Capistrano On Capfile, the place to include more recipes 99

138 Get Your Capistrano On config/deploy.rb, application specific configuration 100

139 Capistrano s Defaults! Your SCM is Subversion! Deployment directory is /u/apps/#{application_name}! User for SCM and SSH is the currently logged-in user! Commands are run with sudo 101

140 Get Your Capistrano On 102

141 Get Your Capistrano On! Capistrano expects a directory structure! Can be created with cap deploy:setup 103

142 The Deployment Lifecycle 104

143 The Deployment Lifecycle Check the prerequisites: 105

144 The Deployment Lifecycle Set up your application for the first time 106

145 The Deployment Lifecycle The initial deployment 1.! Checks the revision from the local machine 2.! Checks out the code on the remote machines 3.! Sets a link called current pointing to the lates release 4.! Runs the migrations 5.! Fires up application servers 107

147 The Deployment Lifecycle Subsequent deployments 1.! Checks the revision from the local machine 2.! Checks out the code on the remote machines 3.! Updates current link 4.! Restarts application servers 109

149 Common Capistrano Tasks Deploy and run migrations Run only the migrations Restart application servers Rollback to the previous release 111

150 Have Your Shell, and Eat It Too 112

151 Invoking any Command 113

152 Deployment Strategies 114

153 Deployment Strategies Direct checkout (from scratch) on the servers 115

154 Deployment Strategies Keep a cached copy of the current SCM head 116

155 Deployment Strategies Check out locally and transfer 117

156 Customizing Capistrano 119

157 Write your own Tasks 120

158 Write your own Tasks 121

159 Namespace your Tasks 122

160 Callbacks Execute a task before another runs Execute a task after another has finished Callbacks are run in the order they re defined 123

161 Useful Variables 124

162 Transactions 125

166 The Rest! Gem dependencies! Support for deploying through gateway servers! Parallel execution on multiple servers! Server setup with deprec gem 129

167 Vlad the Deployer! Simple, Rake-based deployment solution! Comes with defaults for most popular SCM, app and web servers 131

168 Vlad the Deployer Load Configure 132

169 Vlad the Deployer Deployment 133

170 Vlad the Deployer! Simple, sometimes too simple! No callback mechanism, instead the Rake syntax must be used! Very non-verbose output, using --trace recommended 134

171 Chef! System Integration Framework! Written in Ruby inspired by Puppet! Manage multiple nodes! Configuration! Packages! Custom scripts and commands 135

172 Architecture 136

173 Recipes Install and upgrade packages 137

174 Recipes Create and update remote files 138

175 Recipes Create and update remote files 139

176 chef-deploy Enable Use in recipe 140

177 Chef and chef-deploy! In heavy development: sometimes unstable and hard to debug! Be prepared to get your hands dirty! Better suited for configuration than deployment! Community recipes at 141

178 Infrastructure Monitoring 154

179 The two questions of monitoring 155

180 1. Is everything still running? 156

181 2. What are the trends? 157

182 Monit! Process-level monitoring! Checks PID-files, ports, and permissions! Reacts by executing a script and/or alerting! Has a web GUI for administering nodes 158

183 Monit MySQL 159

184 Monit Apache 160

185 Monit Mongrel 161

186 God! Ruby-based configuration! Easily extendible! Can work with event- and poll-based conditions! Comes with less recipes than Monit 162

187 God Mongrel 163

188 Munin! Host-level monitoring! Master periodically asks nodes for local data! Checks system resources and records historical data! Allows to recognize trends and make predictions! Alerting support 164

189 Munin 165

190 Munin 166

191 Munin 167

192 Exception Monitoring! Monitor errors raised by your application! Usually notifying you by ! Good old exception_notification Plugin! Exceptions as a Service 168

193 Hoptoad 169

194 Exceptional 170

195 New Relic! Rails Performance monitoring! Live view into your application 171

196 Other Tools! Nagios! Big Brother! New Relic RPM! FiveRuns! JMX 172

197 Infrastructure Advanced Deployment 173

198 Search 174

199 Search Full text search Can become very slow on big data sets 175

200 Full Text Search Engine Separate Service! Creates full text index! Application queries search daemon! Index update through application or database Possible Engines! Ferret! Sphinx! Solr! Lucene! 176

201 Search Slave Database replication slave! Has complete dataset! Migrates slow search queries from master! Can use different database table engine 177

202 Database Index PostgreSQL Tsearch2! Core since 8.3! Allows to create full text index on multiple columns or arbitrary SQL expressions MySQL MyISAM FULLTEXT index! Only works with MySQL <= 5.0 and MyISAM tables! Full text index on multiple columns 178

203 What to use? Different characteristics! Real-time updates and stale data! Lost updates! Performance! Document content and format! Complexity 179

204 Skalierung

205 does rails scale?

206 what's scaling?

207 Was kann passieren?

208 Server-CPU-Last zu hoch (Bad!)

209 DB-Server-Last zu hoch (Bad!)

210 Zu wenige Connections (Luxusproblem...)

211 Verstopfte Leitungen (eher selten)

212 Volle Festplatten (kill -9 administrator)

213 Festplattendurchsatz! (yeah, right!)

214 viele Probleme, viele Lösungen

215 3 Grundregeln:

216 1. Do not optimize without measurement.

219 Serverlast:

220 Throw Hardware At It

221 Zwei oder mehr App- Server. (Gern auch: VMs)

222 client rails rails rails webserver rails rails rails database Dateisystem public/ rails rails rails

223 Share nothing!

224 Dinge, die üblicherweise shared sind:

225 Sessions (nur bei Rails <2)

226 Caches

227 Hochgeladene Dateien

228 Lösungen:

229 Datenbanken

230 Shared Filesystems

231 memcached

232 Folgefehler: Zu wenige DB-Connections, bzw. zu hohe DB-Last

233 client rails rails rails webserver rails rails rails database Dateisystem public/ rails rails rails

234 Zusätzliche Komplexität: DB-Replikation

235 Back to the lab.

236 'cause I can fix it in the mix

237 Caching!

238 1. Page caching

239 class BlogController < ApplicationController caches_page :list def list Post.find(:all, :order => "created_on desc", :limit => 10) end end

240 2. Action caching

241 caches_action :list

242 3. Fragment caching

243 <strong>my Blog Posts</strong> <% cache do %> <ul> <% for post %> <li><%= link_to post.title, :controller => 'blog', :action => 'show', :id => post %></li> <% end %> </ul> <% end %>

244 def list unless = Post.find(:all,!! :order => 'created_on desc', :limit => 10) end end

245 def list unless = Post.find(:all,!! :order => 'created_on desc', :limit => 10) end end Race condition!

246 4. Model caching

247 Cache invalidation

248 expire_page(:controller => 'blog', :action => 'list') expire_action(:controller => 'blog', :action => 'list') expire_fragment(:controller => 'blog', :action => 'list',! :page => 1)

249 class BlogSweeper < ActionController::Caching::Sweeper observe Post def after_create(post) expire_cache_for(post) end # [...after_update...after_destroy...] private end def expire_cache_for(record) expire_page(:controller => 'blog', :action => 'list') expire_page(:controller => 'blog', :action => 'show', :id => record.id) end

250 Caches müssen auch skalieren

251 Cache-store

252 DB-Server-Last

253 Queries reduzieren

254 Datenbank- Optimierungen

255 Know your Database.

256 Und viele, viele, viele, viele weitere kleine Dinge.

257 You have been warned.

258 Scheduling

259 simpel: cron jobs

260 > crontab -e

261 Rails 3: Über./script/runner kann rake ein komplettes Rails geladen werden. task :foo => :environment rails runner

262 #!/usr/bin/env = GamePlan.find_by_id($ARGV[0]) puts "game_plan not found" exit(5) end

263 Achtung: Gute Cronjobs sind auch aufwändig.

264 DJ (delayed_job) plugin

265 class NewsletterJob < Struct.new(:text, : s) def perform s.each do e NewsletterMailer.deliver_text_to_ (text, e) end end end Delayed::Job.enqueue NewsletterJob.new( 'lorem ipsum...', Customers.find(:all).collect( &: ) )

266 Background Processing 180

267 Problem Long running tasks! Resizing uploaded images! Mailing! Computing an expensive operation! Accessing slow back-ends When running inside request-response-cycle! Blocks user! Blocks Rails instance! Hard to monitor and debug 181

268 Solution Asynchronous processing in the background Message/Queue Scheduler 182

269 Background Processing 183

270 Options Options for message bus:! Database! Amazon SQS! Drb! Memcache! ActiveMQ! Options for background process:! (Ruby) Daemon! Cron job with script/runner! Forked process! Delayed Job / BJ / (Backgroundrb)! run_later!. 184

271 Database/Ruby daemon example 185

272 Cloud Infrastructure 186

273 Cloud Infrastructure Servers come and go! You do not know your servers before deploying! Restarting is the same as introducing a new machine You can t hardcode IPs database.yml 187

274 Solution #1 Query and manually adjust! Servers do not change that often! New nodes probably need manual intervention! Use AWS ElasticIPs to ease the pain Set servers dynamically AWS Elastic IP 188

275 Solution #2 Use a central directory service! A central place to manage your running instances! Instances query the directory and react 189

276 Solution #2 Use a central directory service! A central place to manage your running instances! Instances query the directory and react 190

277 Central Directory Different Implementations! File on S3! SimpleDB! A complete service, capable of monitoring and controlling your instances 191

278 Q & A 192