Mit hotfixes von Carsten Bormann 2011-03-05 + 2013-02-28 Deploying and Monitoring Ruby on Rails A practical guide Mathias Meyer and Jonathan Weiss, 25.05.2009 Peritor GmbH Danke!
Mashed up with... 2011-03-05 Deployment/Scaling/ Caching/Debugging jan krutisch mindmatters gmbh&co. kg jan.krutisch@mindmatters.de http://mindmatters.de
Deployment
Real artists ship!
Komponenten
client (Browser) rails database
client (Browser) Heute: rails thin database mongrel ( unicorn)
mongrel ist schnell
apache ist schneller
client webserver rails Dateisystem public/ database
rails ist nicht threadsafe
client rails webserver rails database Dateisystem public/ rails
webserver rails
webserver rails
oldschool: CGI / SCGI / FastCGI
newschool: http
mongrel versus...
mod_proxy_balancer (in Apache/nginx/lighttpd)
oder ein richtiger proxy
oder ein richtiger balancer
Varnish HAProxy, Squid, etc.
true school: passenger
Apache-Modul
client rails webserver rails database rails Dateisystem public/ mod_passenger
automagisch. it just works.
http://modrails.com
Infrastructure Infrastructure 5
Simple Rails Setup One Rails instance handles all requests Rails is single-threaded: There is only one concurrent request 6
Rails Setup 7
Rails Setup 8
Typical Rails Setup! A load-balancer distributes the incoming requests! Some load-balancers will deliver static requests themselves! Several Rails instances handle all requests! Number of concurrent requests equals number of Rails instances 9
Rails Setup Options 10
Deployment Questions Mongrel? mod_proxy_balancer? Proxy? Apache? Pound? FastCGI? Load-balancer? Nginx? mod_rails? HA-Proxy? Ebb? Reverse Proxy? Swiftiply? Phusion Passenger? Thin? Rails Application Server? Lighttpd? Pen? 11
What we are going to cover today Rails Application Server! FastCGI! Mongrel! mod_rails / Phussion Passenger! JRuby + Glassfish & Co. Proxy/Web Server! Apache2! Nginx! Lighttpd! HA-Proxy 12
FastCGI 13
FastCGI! Protocol to communicate with long-running CGI applications! Usage of either mod_fcgi with Apache 1.3 or mod_fcgi with Lighttpd! Proxy local and remote FastCGI instances! Oldest way of deploying Rails! Deprecated and unstable! Hard to debug (FastCGI protocol) 14
FastCGI 15
FastCGI 16
17
Mongrel 18
Mongrel! Developed by Zed Shaw as an alternative to FastCGI! Complete HTTP-Server that can load arbitrary Ruby-servlets! Built-in Rails support 19
Mongrel 20
Mongrel Apache 2.2 mod_proxy_balancer Lighttpd Nginx HA-Proxy Pound 21
Mongrel Cluster Utility to manage several Mongrel instances 22
Mongrel Cluster Control Mongrels 23
Mongrel and Apache 2.2 Define Mongrel Cluster in Apache 24
Simple Mongrel and Apache 2.2 Redirect all traffic to the Mongel cluster 25
A more complex example! Redirect dynamic requests! Serve static content! Support cached pages! Support maintenance page! Enable client-side caching of images, stylesheets, and JavaScript! Compress output if supported 26
Mongrel! Very robust! Strict HTTP parser! Easy to debug (HTTP!)! Used to be defacto deployment setup with Apache 2.2 and mod_proxy_balancer! Can be a bit difficult to setup (mongrel_cluster, ports, Apache)! Not so easy on mass/virtual hosting 27
mod_rails 28
mod_rails a.k.a Phusion Passenger! Module for Apache 2.2 (and Nginx)! Allows Apache to control Rails instances! Apache starts and stops application instances depending on the application load! Able to run any Rack-compatible Ruby application (Merb, Sinatra & Co.)! Only manages Rails on one host - no remote instances! Combine with HTTP-Proxy / balancing solution 29
Install Phusion Passenger Install Apache module Load and activate in Apache 30
Install Phusion Passenger Install Passenger with nginx (installs custom nginx) Configure nginx 31
Customized Phusion Passenger Control Rails instance number 32
Control Phusion Passenger Restart after deployment: 33
Phusion Passenger One machine 34
Phusion Passenger One machine Multiple machines 35
Phusion Passenger! Ready for production! Makes setup easier on the single machine level! Multiple servers still require load balancer! Suitable for mass-hosting! upcoming standard way of deploying Rails 36
JRuby 37
! Ruby Runtime on the Java Virtual Machine! Implemented in Java and Ruby! Compiles Ruby into Java-bytecode! Integrates with Java code and libraries! Java s promises of native threads and JIT! Allows for Ruby/Rails applications to be packaged as WAR files! WAR files deployable on any J2EE-container: Glassfish, JBoss, Tomcat, Jetty, 38
JRuby on Rails 39
JRuby on Glassfish One machine 40
JRuby on Glassfish One machine Multiple machines 41
Setup JRuby on Glassfish 1.! Download JRuby and Glassfish 2.! From http://blog.headius.com/2008/08/zero-to-production-in-15-minutes.html 42
Warble Configuration Define min/max Rails runtimes 43
Glassfish for development Shrunk-down version of Glassfish v3 - packaged as a gem Can run any Rack-based application Installation:! $ jruby -S gem install glassfish! $ glassfish! There is no Step 3 44
Rails Setup 45
Proxy Options 46
Proxy Requirements! Hide cluster backend from the user! Load-balancer backend instances! Recognize down hosts! Fair scheduler! (Deliver static content) 47
Apache 2.2! Apache 2.2 introduced mod_proxy_balancer! mod_proxy_balancer can speak to multiple backends and balance requests! Apache can acts as a pure proxy or can also serve static files 48
Apache 2.2! Apache 2.2 introduced mod_proxy_balancer! mod_proxy_balancer can speak to multiple backends and balance requests! Apache can acts as a pure proxy or can also serve static files 49
Apache 2.2! Apache 2.2 introduced mod_proxy_balancer! mod_proxy_balancer can speak to multiple backends and balance requests! Apache can acts as a pure proxy or can also serve static files 50
Apache 2.2 Pro! Stable, robust, and mature! Many people know how to work with Apache! Integrates well with other modules (SVN, DAV, Auth, ) Con! Apache can be complicated to configure! The stock Apache is quite resource-hungry compared to pure proxy solutions 51
Nginx From Russia with love! Nginx - popular Russian webserver with good proxy support! Can load-balance multiple backends and deliver static content! Quite popular with Mongrel as the Rails backend! Recent Passenger support 52
Nginx Configuration Simple proxy example Get complete version here: http://brainspl.at/nginx.conf.txt 53
Nginx Pro! Stable, robust, and fast! Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files! Simpler configuration file! Can directly talk to memcached - SSI Con! More documentation would be nice! No equivalent for many Apache modules 54
Lighttpd! Lightweight and fast webserver! Balancing proxy support! Good FastCGI support! Used to be popular until Mongrel came around 55
Lighttpd Configuration Simple proxy example 56
Lighttpd Pro! Fast and lightweight! Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files! Simpler configuration file Con! Unstable for some people! Slow development cycle! More documentation would be nice! Configuration file can be too simple (virtual host aliasing)! No equivalent for many Apache modules! (No Passenger support) 57
HA-Proxy! HAProxy reliable, high performance TCP/HTTP load balancer! Proxying and content inspection! No content serving, just a proxy! Mature proxy module (fair scheduler)! ACL support See also similar Pound and Pen 58
HAProxy Simple proxy example 59
HAProxy Pro! Mature, stable, robust, and fast! TCP and HTTP balancing Con! Few Rails examples! Usually not needed in a Rails setup 60
Recommended Setups 61
Small Site Recommendation Apache 2.2 with mod_rails / Phusion Passenger 62
Medium Site Recommendation! Apache/Nginx as the frontend proxy! Passenger/mod_rails as the backend! Deliver static files with Apache 63
Large Rails Setup Recommendation! Redundant load-balancer! Redundant proxy / web! Passenger/mod_rails 64
Heavy Static Files Recommendation! Deliver static assets through separate web server farm! Passenger/mod_rails 65
Java Shop Recommendation! Deliver a Rails-WAR file and you are done! Integrate with existing Java landscape and infrastructure 66
67
Application Server Handler 68
Application Server Handler 69
Application Server Handler 70
Rack 71
Rack Handler 72
Rack 73
Rack Middleware 74
Rack Middleware! Authentication (HTTP, OpenID)! Sessions! Routing! Caching! Logging! Debugging 75
Support Frameworks! Rails! Merb! Camping! Sinatra! Mack! Maveric! Halcyon! Ramaze! Rum! Vintage! Waves Web Server! CGI! FastCGI! Mongrel! Phusion Passenger! JRuby Servlets! WEBrick! Ebb! Thin! LiteSpeed! Swiftcore! Unicorn 76
Remarks 77
Ruby Enterprise Edition! Copy-On-Write patches to Ruby 1.8! Saves memory when spawning several Rails instances! Used by Phusion Passenger if available Dead: 1.9.3 2.0.0 78
Thin, Ebb, Evented Mongrel & Co.! Alternatives to Mongrel! Claim to be faster, lighter, and what have you! Rendering Hello World is usually not your bottleneck Stick with stable and robust Mongrel, or better yet, with Passenger Might be somewhat dated 2011-03-05 79
deploy-prozess
prozesse (mongrels) müssen neugestartet werden.
Bei Passenger: touch tmp/restart.txt
capistrano
> cap deploy:migrations
konfiguration in rubydateien. Yay!
set :application, "name" set :repository, "http://kenn.i.net/svn/trunk/name" role :app, "mog.i.net" role :web, "mog.i.net" role :db, "mog.i.net", :primary => true require 'lib/cap/backgroundrb_recipes' after 'deploy:update_code', :set_symlinks task :set_symlinks do run "ln -s #{shared_path}/config/database.yml #{release_path}/config/ database.yml" run "ln -s #{shared_path}/data #{release_path}/public/data" end
AWE13 Gruppe/04 danke!
AWE13 Gruppe/04 danke!
Erweitern um eigene "Recipes"
Kommandos werden parallel auf allen Servern ausgeführt
per SSH
Unterstützung diverser SCM-Systeme
Erlaubt (relativ) einfache rollbacks, jeder deploystand wird abgelegt.
Quasi-Standard
Nicht auf Rails beschränkt!
Infrastructure Deployment 80
Deployment Options 82
Deployment Options 83
Deployment Options 84
Deployment Options 85
What does Capistrano do? 86
Capistrano Deployment Cycle 87
Requirements 88
What doesn t Capistrano do?! Plan your initial server setup! Configure basic services 89
Basic Ingredients! The cap command! Variables! Roles! Tasks! Namespaces 90
Basic Ingredients - cap Your one-stop deployment shop 91
Basic Ingredients - Variables! Configure basic project information! Override Capistrano s default assumptions! Once set, variables are available globally! Defined using the set method 92
Basic Ingredients - Roles! Define types of servers! Default roles! :www! :app! :db! All can point to the same server! But all three must be defined! At least one database server needs to be primary 93
Basic Ingredients - Roles Define custom roles as you please Can be reused when defining tasks 94
Basic Ingredients - Tasks! Define an atomic set of actions for Capistrano! Can be called from the command line! Or other tasks 95
Basic Ingredients - Tasks To find all the tasks available in your project, use 96
Basic Ingredients - Namespaces Group tasks together logically Namespaces and tasks are separated with : 97
Get Your Capistrano On 98
Get Your Capistrano On Capfile, the place to include more recipes 99
Get Your Capistrano On config/deploy.rb, application specific configuration 100
Capistrano s Defaults! Your SCM is Subversion! Deployment directory is /u/apps/#{application_name}! User for SCM and SSH is the currently logged-in user! Commands are run with sudo 101
Get Your Capistrano On 102
Get Your Capistrano On! Capistrano expects a directory structure! Can be created with cap deploy:setup 103
The Deployment Lifecycle 104
The Deployment Lifecycle Check the prerequisites: 105
The Deployment Lifecycle Set up your application for the first time 106
The Deployment Lifecycle The initial deployment 1.! Checks the revision from the local machine 2.! Checks out the code on the remote machines 3.! Sets a link called current pointing to the lates release 4.! Runs the migrations 5.! Fires up application servers 107
The Deployment Lifecycle 108
The Deployment Lifecycle Subsequent deployments 1.! Checks the revision from the local machine 2.! Checks out the code on the remote machines 3.! Updates current link 4.! Restarts application servers 109
The Deployment Lifecycle 110
Common Capistrano Tasks Deploy and run migrations Run only the migrations Restart application servers Rollback to the previous release 111
Have Your Shell, and Eat It Too 112
Invoking any Command 113
Deployment Strategies 114
Deployment Strategies Direct checkout (from scratch) on the servers 115
Deployment Strategies Keep a cached copy of the current SCM head 116
Deployment Strategies Check out locally and transfer 117
Customizing Capistrano 119
Write your own Tasks 120
Write your own Tasks 121
Namespace your Tasks 122
Callbacks Execute a task before another runs Execute a task after another has finished Callbacks are run in the order they re defined 123
Useful Variables 124
Transactions 125
Transactions 126
Transactions 127
Transactions 128
The Rest! Gem dependencies! Support for deploying through gateway servers! Parallel execution on multiple servers! Server setup with deprec gem 129
Vlad the Deployer! Simple, Rake-based deployment solution! Comes with defaults for most popular SCM, app and web servers 131
Vlad the Deployer Load Configure 132
Vlad the Deployer Deployment 133
Vlad the Deployer! Simple, sometimes too simple! No callback mechanism, instead the Rake syntax must be used! Very non-verbose output, using --trace recommended 134
Chef! System Integration Framework! Written in Ruby inspired by Puppet! Manage multiple nodes! Configuration! Packages! Custom scripts and commands 135
Architecture 136
Recipes Install and upgrade packages 137
Recipes Create and update remote files 138
Recipes Create and update remote files 139
chef-deploy Enable Use in recipe 140
Chef and chef-deploy! In heavy development: sometimes unstable and hard to debug! Be prepared to get your hands dirty! Better suited for configuration than deployment! Community recipes at http://github.com/opscode/cookbooks 141
Infrastructure Monitoring 154
The two questions of monitoring 155
1. Is everything still running? 156
2. What are the trends? 157
Monit! Process-level monitoring! Checks PID-files, ports, and permissions! Reacts by executing a script and/or alerting! Has a web GUI for administering nodes 158
Monit MySQL 159
Monit Apache 160
Monit Mongrel 161
God! Ruby-based configuration! Easily extendible! Can work with event- and poll-based conditions! Comes with less recipes than Monit 162
God Mongrel 163
Munin! Host-level monitoring! Master periodically asks nodes for local data! Checks system resources and records historical data! Allows to recognize trends and make predictions! Alerting support 164
Munin 165
Munin 166
Munin 167
Exception Monitoring! Monitor errors raised by your application! Usually notifying you by email! Good old exception_notification Plugin! Exceptions as a Service 168
Hoptoad 169
Exceptional 170
New Relic! Rails Performance monitoring! Live view into your application 171
Other Tools! Nagios! Big Brother! New Relic RPM! FiveRuns! JMX 172
Infrastructure Advanced Deployment 173
Search 174
Search Full text search Can become very slow on big data sets 175
Full Text Search Engine Separate Service! Creates full text index! Application queries search daemon! Index update through application or database Possible Engines! Ferret! Sphinx! Solr! Lucene! 176
Search Slave Database replication slave! Has complete dataset! Migrates slow search queries from master! Can use different database table engine 177
Database Index PostgreSQL Tsearch2! Core since 8.3! Allows to create full text index on multiple columns or arbitrary SQL expressions MySQL MyISAM FULLTEXT index! Only works with MySQL <= 5.0 and MyISAM tables! Full text index on multiple columns 178
What to use? Different characteristics! Real-time updates and stale data! Lost updates! Performance! Document content and format! Complexity 179
Skalierung
does rails scale?
what's scaling?
Was kann passieren?
Server-CPU-Last zu hoch (Bad!)
DB-Server-Last zu hoch (Bad!)
Zu wenige Connections (Luxusproblem...)
Verstopfte Leitungen (eher selten)
Volle Festplatten (kill -9 administrator)
Festplattendurchsatz! (yeah, right!)
viele Probleme, viele Lösungen
3 Grundregeln:
1. Do not optimize without measurement.
2. Do not optimize without measurement.
3. Do not optimize without measurement.
Serverlast:
Throw Hardware At It
Zwei oder mehr App- Server. (Gern auch: VMs)
client rails rails rails webserver rails rails rails database Dateisystem public/ rails rails rails
Share nothing!
Dinge, die üblicherweise shared sind:
Sessions (nur bei Rails <2)
Caches
Hochgeladene Dateien
Lösungen:
Datenbanken
Shared Filesystems
memcached
Folgefehler: Zu wenige DB-Connections, bzw. zu hohe DB-Last
client rails rails rails webserver rails rails rails database Dateisystem public/ rails rails rails
Zusätzliche Komplexität: DB-Replikation
Back to the lab.
'cause I can fix it in the mix
Caching!
1. Page caching
class BlogController < ApplicationController caches_page :list def list Post.find(:all, :order => "created_on desc", :limit => 10) end end
2. Action caching
caches_action :list
3. Fragment caching
<strong>my Blog Posts</strong> <% cache do %> <ul> <% for post in @posts %> <li><%= link_to post.title, :controller => 'blog', :action => 'show', :id => post %></li> <% end %> </ul> <% end %>
def list unless read_fragment({}) @post = Post.find(:all,!! :order => 'created_on desc', :limit => 10) end end
def list unless read_fragment({}) @post = Post.find(:all,!! :order => 'created_on desc', :limit => 10) end end Race condition!
4. Model caching
Cache invalidation
expire_page(:controller => 'blog', :action => 'list') expire_action(:controller => 'blog', :action => 'list') expire_fragment(:controller => 'blog', :action => 'list',! :page => 1)
class BlogSweeper < ActionController::Caching::Sweeper observe Post def after_create(post) expire_cache_for(post) end # [...after_update...after_destroy...] private end def expire_cache_for(record) expire_page(:controller => 'blog', :action => 'list') expire_page(:controller => 'blog', :action => 'show', :id => record.id) end
Caches müssen auch skalieren
Cache-store
DB-Server-Last
Queries reduzieren
Datenbank- Optimierungen
Know your Database.
Und viele, viele, viele, viele weitere kleine Dinge.
You have been warned.
Scheduling
simpel: cron jobs
> crontab -e
Rails 3: Über./script/runner kann rake ein komplettes Rails geladen werden. task :foo => :environment rails runner
#!/usr/bin/env ruby./script/runner @game_plan = GamePlan.find_by_id($ARGV[0]) if @game_plan.nil? puts "game_plan not found" exit(5) end
Achtung: Gute Cronjobs sind auch aufwändig.
DJ (delayed_job) plugin
class NewsletterJob < Struct.new(:text, :emails) def perform emails.each do e NewsletterMailer.deliver_text_to_email(text, e) end end end Delayed::Job.enqueue NewsletterJob.new( 'lorem ipsum...', Customers.find(:all).collect( &:email ) )
Background Processing 180
Problem Long running tasks! Resizing uploaded images! Mailing! Computing an expensive operation! Accessing slow back-ends When running inside request-response-cycle! Blocks user! Blocks Rails instance! Hard to monitor and debug 181
Solution Asynchronous processing in the background Message/Queue Scheduler 182
Background Processing 183
Options Options for message bus:! Database! Amazon SQS! Drb! Memcache! ActiveMQ! Options for background process:! (Ruby) Daemon! Cron job with script/runner! Forked process! Delayed Job / BJ / (Backgroundrb)! run_later!. 184
Database/Ruby daemon example 185
Cloud Infrastructure 186
Cloud Infrastructure Servers come and go! You do not know your servers before deploying! Restarting is the same as introducing a new machine You can t hardcode IPs database.yml 187
Solution #1 Query and manually adjust! Servers do not change that often! New nodes probably need manual intervention! Use AWS ElasticIPs to ease the pain Set servers dynamically AWS Elastic IP 188
Solution #2 Use a central directory service! A central place to manage your running instances! Instances query the directory and react 189
Solution #2 Use a central directory service! A central place to manage your running instances! Instances query the directory and react 190
Central Directory Different Implementations! File on S3! SimpleDB! A complete service, capable of monitoring and controlling your instances 191
Q & A 192
Peritor GmbH Teutonenstraße 16 14129 Berlin Telefon: +49 (0)30 69 20 09 84 0 Telefax: +49 (0)30 69 20 09 84 9 Internet: www.peritor.com E-Mail: kontakt@peritor.com!peritor GmbH - Alle Rechte vorbehalten 193 193