Drupal Memcached Nginx

Drupal Memcached Nginx

Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do)

Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do)

Technology overview Nginx PHP-FPM Memcached Xhprof Scrapy Siege

Nginx 1000 concurrent connection == normal Serves as a connection 'router' Advanced configuration (variables, control structures,...) Extend with modules http://nginx.org/

PHP-FPM FastCGI Process Manager for PHP Werkt adhv configuratie 'pool' Elk item heeft eigen configuratie Unix socket of TCP poort per entry in pool Chroot http://php-fpm.org/

Memcached Key value store Distributed Key max. 255 chars http://memcached.org/

XHProf Profiler Time spend in each function (inclusive/exclusive wall time) Drill down in functions to pinpoint problems

Scrapy Webscraper (Spider) Sites indexeren Spamlijsten aanleggen Xpath http://scrapy.org/

Siege Advanced 'Apache Benchmark' Crawl list of URL's Output: URL Responscode Time siege -c 100 -t 5m -b -i -f url-list.txt http://joedog.org/siege-home/

Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do)

Drupal optimization Optimize application Reduce server load Tune your stack

Drupal optimization Optimize application Reduce server load Tune your stack

Optimize application (1) First Turn pagecaching OFF Turn poormanscron OFF (bad UX) Turn on slow query log (Devel) while developing

Optimize application (2) Crawl site with Speedy Visit all URL's once with Siege Pipe output to file convert to CSV Response code 404's? Internal broken links 302? Extra hit on server! 500? Upstream server is down Tip: tail -f on errorlogs

Optimize application (3) Analyze slow pages XHProf Add caching where applicable (Block, Entity, Views, Webservice calls, ) Turn on php-fpm slowlog on test/production

Drupal optimization Optimize application Reduce server load Tune your stack

Reduce server load Minify Js Aggregate JS & CSS Turn ON pagecaching Correct cache headers CSS Sprites, use repeat-x/y (reduce size) Consider using CDN (CloudFront 0.12$/Gb)

Drupal optimization Optimize application Reduce server load Tune your stack

Tune your stack - Memcached Download & compile Memcached (daemon) Configure, make, make install Start: :~# memcached -u www-data -s /var/run/memcached.sock Use Unix socket if possible (local)

Tune your stack Memcached (2) Install php bindings Apt-get install php5-memcached Enable in php.ini Alter settings.php $conf['cache_backends'][] = './sites/all/modules/contrib/memcache/memcache.inc'; $conf['cache_default_class'] = 'MemCacheDrupal'; $conf['memcache_servers'] = array( ); 'unix:///var/run/memcached.sock' => 'default',

Tune your stack Nginx Compile from source! Check out perusio's excellent scripts Fetch required modules Create a compile script Create vhosts upstream phpcgi { server unix:/var/run/php-fpm.sock; }

Tune your stack PHP-FPM :~# apt-get install php5-fpm Create entry in /etc/php5/fpm/pool.d listen = /var/run/php-fpm.sock slowlog = log/$pool.log.slow request_slowlog_timeout = 4s /etc/init.d/php5-fpm start

Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do)

Future Access memcached from nginx Write new memcached module (hack looks promising) Use ngx_http_enhanced_memcached_module (support headers, key hashing) Soft expiration? server-side locking (stampede protection)? Server side includes Puppet

Drupal Memcached Nginx 1

Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do) 2

Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do) 3

Technology overview Nginx PHP-FPM Memcached Xhprof Scrapy Siege 4

Nginx 1000 concurrent connection == normal Serves as a connection 'router' Advanced configuration (variables, control structures,...) Extend with modules http://nginx.org/ 5

PHP-FPM FastCGI Process Manager for PHP Werkt adhv configuratie 'pool' Elk item heeft eigen configuratie Unix socket of TCP poort per entry in pool Chroot http://php-fpm.org/ 6

Memcached Key value store Distributed Key max. 255 chars http://memcached.org/ 7 Socket forwarden naar poort: sudo socat -v UNIX-CONNECT:/tmp/memcache TCP- LISTEN:6666,reuseaddr

XHProf Profiler Time spend in each function (inclusive/exclusive wall time) Drill down in functions to pinpoint problems 8 Install and usage: http://techportal.ibuildings.com/2009/12/01/profilingwith-xhprof/

Scrapy Webscraper (Spider) Sites indexeren Spamlijsten aanleggen Xpath http://scrapy.org/ 9 Spider.py: from scrapy.selector import HtmlXPathSelector from scrapy.spider import BaseSpider from scrapy.http import Request DOMAIN = 'localhost' URL = 'http://%s' % DOMAIN class MySpider(BaseSpider): name = DOMAIN allowed_domains = [DOMAIN] start_urls = [ URL ] def parse(self, response): hxs = HtmlXPathSelector(response) for url in hxs.select('//a/@href').extract(): if not url.startswith('http://'): url= URL + url print url yield Request(url, callback=self.parse) scrapy runspider spider.py > urls.out

Siege Advanced 'Apache Benchmark' Crawl list of URL's Output: URL Responscode Time siege -c 100 -t 5m -b -i -f url-list.txt http://joedog.org/siege-home/ 10 Siege -c 100 -t 5m -b -i -f url-lijst.txt c => Concurrent users t => Tijd (5 minuten) B => Geen delay tussen requests i => Internet random select uit lijst f => File met URL's

Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do) 11

Drupal optimization Optimize application Reduce server load Tune your stack 12

Drupal optimization Optimize application Reduce server load Tune your stack 13

Optimize application (1) First Turn pagecaching OFF Turn poormanscron OFF (bad UX) Turn on slow query log (Devel) while developing 14

Optimize application (2) Crawl site with Speedy Visit all URL's once with Siege Pipe output to file convert to CSV Response code 404's? Internal broken links 302? Extra hit on server! 500? Upstream server is down Tip: tail -f on errorlogs 15

Optimize application (3) Analyze slow pages XHProf Add caching where applicable (Block, Entity, Views, Webservice calls, ) Turn on php-fpm slowlog on test/production 16

Drupal optimization Optimize application Reduce server load Tune your stack 17

Reduce server load Minify Js Aggregate JS & CSS Turn ON pagecaching Correct cache headers CSS Sprites, use repeat-x/y (reduce size) Consider using CDN (CloudFront 0.12$/Gb) 18

Drupal optimization Optimize application Reduce server load Tune your stack 19

Tune your stack - Memcached Download & compile Memcached (daemon) Configure, make, make install Start: :~# memcached -u www-data -s /var/run/memcached.sock Use Unix socket if possible (local) 20 -Create socket first: sudo touch /var/run/memcached.pid -Settings

Tune your stack Memcached (2) Install php bindings Apt-get install php5-memcached Enable in php.ini Alter settings.php $conf['cache_backends'][] = './sites/all/modules/contrib/memcache/memcache.inc'; $conf['cache_default_class'] = 'MemCacheDrupal'; $conf['memcache_servers'] = array( ); 'unix:///var/run/memcached.sock' => 'default', 21

Tune your stack Nginx Compile from source! Check out perusio's excellent scripts Fetch required modules Create a compile script Create vhosts upstream phpcgi { server unix:/var/run/php-fpm.sock; } 22 Perusio's Nginx-Drupal scripts: https://github.com/perusio/drupal-with-nginx My build script: cd./nginx-1.3.0./configure \ --prefix=/opt/nginx \ --add-module=../ngx_devel_kit \ --add-module=../set-misc-nginx-module \ --add-module=../nginx-upload-progress-module \ --add-module=../nginx-upstream-fair \ --add-module=../ngx_http_enhanced_memcached_module \ --with-http_realip_module \ --with-http_ssl_module \ --with-file-aio \ --with-http_flv_module \ --with-http_mp4_module \ make make install cd /opt/nginx/conf/ rm./* -R cp /opt/nginx-sources/conf/*./ -R

Tune your stack PHP-FPM :~# apt-get install php5-fpm Create entry in /etc/php5/fpm/pool.d listen = /var/run/php-fpm.sock slowlog = log/$pool.log.slow request_slowlog_timeout = 4s /etc/init.d/php5-fpm start 23

Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do) 24

Future Access memcached from nginx Write new memcached module (hack looks promising) Use ngx_http_enhanced_memcached_module (support headers, key hashing) Soft expiration? server-side locking (stampede protection)? Server side includes Puppet 25

26