Drupal Memcached Nginx

Transcription

1 Drupal Memcached Nginx

2 Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do)

3 Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do)

4 Technology overview Nginx PHP-FPM Memcached Xhprof Scrapy Siege

5 Nginx 1000 concurrent connection == normal Serves as a connection 'router' Advanced configuration (variables, control structures,...) Extend with modules

6 PHP-FPM FastCGI Process Manager for PHP Werkt adhv configuratie 'pool' Elk item heeft eigen configuratie Unix socket of TCP poort per entry in pool Chroot

7 Memcached Key value store Distributed Key max. 255 chars

8 XHProf Profiler Time spend in each function (inclusive/exclusive wall time) Drill down in functions to pinpoint problems

9 Scrapy Webscraper (Spider) Sites indexeren Spamlijsten aanleggen Xpath

10 Siege Advanced 'Apache Benchmark' Crawl list of URL's Output: URL Responscode Time siege -c 100 -t 5m -b -i -f url-list.txt

11 Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do)

12 Drupal optimization Optimize application Reduce server load Tune your stack

13 Drupal optimization Optimize application Reduce server load Tune your stack

14 Optimize application (1) First Turn pagecaching OFF Turn poormanscron OFF (bad UX) Turn on slow query log (Devel) while developing

15 Optimize application (2) Crawl site with Speedy Visit all URL's once with Siege Pipe output to file convert to CSV Response code 404's? Internal broken links 302? Extra hit on server! 500? Upstream server is down Tip: tail -f on errorlogs

16 Optimize application (3) Analyze slow pages XHProf Add caching where applicable (Block, Entity, Views, Webservice calls, ) Turn on php-fpm slowlog on test/production

17 Drupal optimization Optimize application Reduce server load Tune your stack

18 Reduce server load Minify Js Aggregate JS & CSS Turn ON pagecaching Correct cache headers CSS Sprites, use repeat-x/y (reduce size) Consider using CDN (CloudFront 0.12$/Gb)

19 Drupal optimization Optimize application Reduce server load Tune your stack

20 Tune your stack - Memcached Download & compile Memcached (daemon) Configure, make, make install Start: :~# memcached -u www-data -s /var/run/memcached.sock Use Unix socket if possible (local)

21 Tune your stack Memcached (2) Install php bindings Apt-get install php5-memcached Enable in php.ini Alter settings.php $conf['cache_backends'][] = './sites/all/modules/contrib/memcache/memcache.inc'; $conf['cache_default_class'] = 'MemCacheDrupal'; $conf['memcache_servers'] = array( ); 'unix:///var/run/memcached.sock' => 'default',

22 Tune your stack Nginx Compile from source! Check out perusio's excellent scripts Fetch required modules Create a compile script Create vhosts upstream phpcgi { server unix:/var/run/php-fpm.sock; }

23 Tune your stack PHP-FPM :~# apt-get install php5-fpm Create entry in /etc/php5/fpm/pool.d listen = /var/run/php-fpm.sock slowlog = log/$pool.log.slow request_slowlog_timeout = 4s /etc/init.d/php5-fpm start

24 Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do)

25 Future Access memcached from nginx Write new memcached module (hack looks promising) Use ngx_http_enhanced_memcached_module (support headers, key hashing) Soft expiration? server-side locking (stampede protection)? Server side includes Puppet

26

27 Drupal Memcached Nginx 1

28 Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do) 2

29 Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do) 3

30 Technology overview Nginx PHP-FPM Memcached Xhprof Scrapy Siege 4

31 Nginx 1000 concurrent connection == normal Serves as a connection 'router' Advanced configuration (variables, control structures,...) Extend with modules 5

32 PHP-FPM FastCGI Process Manager for PHP Werkt adhv configuratie 'pool' Elk item heeft eigen configuratie Unix socket of TCP poort per entry in pool Chroot 6

33 Memcached Key value store Distributed Key max. 255 chars 7 Socket forwarden naar poort: sudo socat -v UNIX-CONNECT:/tmp/memcache TCP- LISTEN:6666,reuseaddr

34 XHProf Profiler Time spend in each function (inclusive/exclusive wall time) Drill down in functions to pinpoint problems 8 Install and usage:

35 Scrapy Webscraper (Spider) Sites indexeren Spamlijsten aanleggen Xpath 9 Spider.py: from scrapy.selector import HtmlXPathSelector from scrapy.spider import BaseSpider from scrapy.http import Request DOMAIN = 'localhost' URL = ' % DOMAIN class MySpider(BaseSpider): name = DOMAIN allowed_domains = [DOMAIN] start_urls = [ URL ] def parse(self, response): hxs = HtmlXPathSelector(response) for url in hxs.select('//a/@href').extract(): if not url.startswith(' url= URL + url print url yield Request(url, callback=self.parse) scrapy runspider spider.py > urls.out

36 Siege Advanced 'Apache Benchmark' Crawl list of URL's Output: URL Responscode Time siege -c 100 -t 5m -b -i -f url-list.txt 10 Siege -c 100 -t 5m -b -i -f url-lijst.txt c => Concurrent users t => Tijd (5 minuten) B => Geen delay tussen requests i => Internet random select uit lijst f => File met URL's

37 Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do) 11

38 Drupal optimization Optimize application Reduce server load Tune your stack 12

39 Drupal optimization Optimize application Reduce server load Tune your stack 13

40 Optimize application (1) First Turn pagecaching OFF Turn poormanscron OFF (bad UX) Turn on slow query log (Devel) while developing 14

41 Optimize application (2) Crawl site with Speedy Visit all URL's once with Siege Pipe output to file convert to CSV Response code 404's? Internal broken links 302? Extra hit on server! 500? Upstream server is down Tip: tail -f on errorlogs 15

42 Optimize application (3) Analyze slow pages XHProf Add caching where applicable (Block, Entity, Views, Webservice calls, ) Turn on php-fpm slowlog on test/production 16

43 Drupal optimization Optimize application Reduce server load Tune your stack 17

44 Reduce server load Minify Js Aggregate JS & CSS Turn ON pagecaching Correct cache headers CSS Sprites, use repeat-x/y (reduce size) Consider using CDN (CloudFront 0.12$/Gb) 18

45 Drupal optimization Optimize application Reduce server load Tune your stack 19

46 Tune your stack - Memcached Download & compile Memcached (daemon) Configure, make, make install Start: :~# memcached -u www-data -s /var/run/memcached.sock Use Unix socket if possible (local) 20 -Create socket first: sudo touch /var/run/memcached.pid -Settings

47 Tune your stack Memcached (2) Install php bindings Apt-get install php5-memcached Enable in php.ini Alter settings.php $conf['cache_backends'][] = './sites/all/modules/contrib/memcache/memcache.inc'; $conf['cache_default_class'] = 'MemCacheDrupal'; $conf['memcache_servers'] = array( ); 'unix:///var/run/memcached.sock' => 'default', 21

48 Tune your stack Nginx Compile from source! Check out perusio's excellent scripts Fetch required modules Create a compile script Create vhosts upstream phpcgi { server unix:/var/run/php-fpm.sock; } 22 Perusio's Nginx-Drupal scripts: My build script: cd./nginx /configure \ --prefix=/opt/nginx \ --add-module=../ngx_devel_kit \ --add-module=../set-misc-nginx-module \ --add-module=../nginx-upload-progress-module \ --add-module=../nginx-upstream-fair \ --add-module=../ngx_http_enhanced_memcached_module \ --with-http_realip_module \ --with-http_ssl_module \ --with-file-aio \ --with-http_flv_module \ --with-http_mp4_module \ make make install cd /opt/nginx/conf/ rm./* -R cp /opt/nginx-sources/conf/*./ -R

49 Tune your stack PHP-FPM :~# apt-get install php5-fpm Create entry in /etc/php5/fpm/pool.d listen = /var/run/php-fpm.sock slowlog = log/$pool.log.slow request_slowlog_timeout = 4s /etc/init.d/php5-fpm start 23

50 Drupal, Memcached &Nginx Technology overview Drupal optimization Future (To Do) 24

51 Future Access memcached from nginx Write new memcached module (hack looks promising) Use ngx_http_enhanced_memcached_module (support headers, key hashing) Soft expiration? server-side locking (stampede protection)? Server side includes Puppet 25

52 26