Optimizing your Magento store for heavy API access 1.0 June 2013
About This Guide This guide is for anyone who is looking to optimise the API link speed between a Magento installation and Brightpearl. In this guide we will offer practical advice and settings for MySQL, PHP and Magento itself. In general most VPS/Cloud or dedicated hosting companies offer both managed and unmanaged hosting services for clients. If you're managing your server directly then you will have access to change and update settings as suggested. If your server is fully managed then you may wish to request these changes are made by your hosting provider. In general we are assuming that you are running cpanel on a Linux distribution and that you are running your Magento installation as an account on this cpanel/linux environment. If you're hosting environment is non standard or distributed then settings suggested in this guide may not apply or be appropriate. Suggestions in this guide are not to be taken as definitive and due consideration should be given the repercussions of changes on other installed software or services. Background Magento is a highly configurable shopping cart built on PHP and MySQL. It is known to offer a great deal of configurability allowing a high level of customisation. It is also known however to suffer from performance issues if poorly configured which can affect not only front end user performance but also affect internal services such as the API interface. Brightpearl utilises the Magento API to perform its communication to Magento. As this is the case it is important to consider API performance to ensure that the connection between the two systems remains stable and trouble free. Using the Magento API means that the servers Apache service, PHP services along with mysql services are all put under load. For that reason optimisation of these 3 critical components is vital. How to update your Magento system If you are not a technical expert, please do not attempt to make the changes to the server yourself. You may take your store offline, and it may be hard to recover, even by professionals. Brightpearl cannot restore your Magento installation or Magento server to a previous state; your Magento store is your own property and access by Brightpearl is not required at any stage of the integration.
Hosting Recommendations Due consideration should be given to the specification of the server (dedicated or virtualised) that you intend to run Magento on. In general the higher the specification a server is will ultimately result in performance improvements. Of course a business decision needs to be made to decide at what price point is appropriate and then obtain the best package possible within that price range. For small to medium size Magento stores of product catalogue size of up to around 20,000 items an entry dedicated server or a similarly specified cloud/vps hosting package is recommended. An example entry level dedicated server specification is as follows: Processor Cores Memory Hard Drives Hot Swappable Drives SSD Available Bandwidth Quad Core Xeon 3.1GHz 4 Core 8GB RAM Network Uptime Guarantee 100% Control Panel Options 2 x 250GB SATA-III RAID1 UNLIMITED Plesk / cpanel Servers below this specification are not guaranteed to cope with the demands placed on it by a Brightpearl integration. Typically we have found that hosting of the above specification costs from 100 / $150 per month upwards. Wherever possible the use of SSD local disks is highly recommended Server Loading How many sites run on your server? Logically the more sites & other services that are being run on a single server the more load that server is under at any given point in time and therefore the slower your Magento store and API may run. Wherever possible we highly recommend the use of one dedicated server for one business critical Magento site/installation. This approach minimises the risk of timeouts and allows for easier debugging of problems, it is also appropriate if you are required to run a PCI DSS validated server.
Apache Optimization 1. Enable module mod_fcgid for Apache What does this do? mod_fcgid is a high performance alternative to mod_cgi or mod_cgid, which starts a sufficient number instances of the CGI program to handle concurrent requests, and these programs remain running to handle further incoming requests. It is favored by the PHP developers, for example, as a preferred alternative to running mod_php in-process, delivering very similar performance. (standard module for Apache version >= 2.2 ) This module can be enabled via EasyApache in the exhaustive options list as below. Build, compile and restart Apache Additional configuration settings for mod_fcgid Include the following configuration lines in configuration file http.conf (accessed from the command line or from Cpanel -> Service Configuration -> Apache Configuration -> Include Editor -> Pre Main Include -> I wish to edit the Pre Main configuration include file for: All Versions) FcgidMaxRequestLen 20971520 FcgidIOTimeout 300 FcgidMaxRequestsPerProcess 5000 FcgidMaxProcesses 20 FcgidMinProcessesPerClass 0 FcgidIdleTimeout 20 FcgidIdleScanInterval 10 FcgidBusyTimeout 20 FcgidBusyScanInterval 10 FcgidErrorScanInterval 5 FcgidZombieScanInterval 5 FcgidProcessLifeTime 10 Example:
Note: for cpanel do not edit the http.conf file directly as this will be overwritten on any cpanel upgrade or Apache rebuild.
2. Enable mod_deflate and mod_headers via command line or EasyApache What does this do? The mod_deflate module provides the DEFLATE output filter that allows output from your server to be compressed before being sent to the client over the network. This reduces bandwidth load on your server. and Build, compile and restart Apache Go to file.htaccess on your Magento site and change deflate section to following: <IfModule mod_deflate.c> ############################################ ## enable apache served files compression ## http://developer.yahoo.com/performance/rules.html#gzip # Insert filter on all content SetOutputFilter DEFLATE # Insert filter on selected content types only AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript # Netscape 4.x has some problems... BrowserMatch ^Mozilla/4 gzip-only-text/html # Netscape 4.06-4.08 have some more problems BrowserMatch ^Mozilla/4\.0[678] no-gzip # MSIE masquerades as Netscape, but it is fine BrowserMatch \bmsie!no-gzip!gzip-only-text/html # Don t compress images
SetEnvIfNoCase Request_URI \.(?:gif jpe?g png)$ no-gzip dontvary vary # Make sure proxies don t deliver the wrong content Header append Vary User-Agent env=!dont-vary </IfModule> Your Magento site should now be passing content in gzipped compressed form. Test here: http://www.whatsmyip.org/http-compression-test/
3. Enable module mod_expires via command line or EasyApache What does this do? This module controls the setting of the Expires HTTP header and the max-age directive of the Cache-Control HTTP header in server responses. The expiration date can set to be relative to either the time the source file was last modified, or to the time of the client access. These HTTP headers are an instruction to the client about the document's validity and persistence. If cached, the document may be fetched from the cache rather than from the source until this time has passed. After that, the cache copy is considered "expired" and invalid, and a new copy must be obtained from the source. Build, compile and restart Apache Go to file.htaccess on your Magento site and change deflate section to following: ############################################ ## Add default Expires header ## http://developer.yahoo.com/performance/rules.html#expires ExpiresActive On ExpiresByType image/gif "access plus 30 days" ExpiresByType image/jpg "access plus 30 days" ExpiresByType image/jpeg "access plus 30 days" ExpiresByType image/png "access plus 30 days" ExpiresByType image/x-icon "access plus 30 days" ExpiresByType text/css "access plus 30 days" ExpiresByType application/x-javascript "access plus 30 days" </IfModule> Test here https://developers.google.com/speed/pagespeed/
4. Additional Apache optimisations.optimizing in configuration file http.conf ( from command line or from Cpanel -> Service Configuration -> Apache Configuration -> Global Configuration Turn Directive `KeepAlive` to 'on' Change the following values StartServers 50 MinSpareServers 15 MaxSpareServers 30 MaxClients 225 MaxRequestsPerChild 4000
PHP optimization 1. Install a PHP accelerator/cache Install APC, Eaccelerator, Zend Accelerator or Xcache on your server or contact with your hosting provider to install these PHP Cache Systems. 2. Update PHP configuration either in php.ini (usually in directory /etc) or via cpanel GUI where applicable. realpath_cache_size=1m realpath_cache_ttl=86400 memory_limit = 128M post_max_size = 20M upload_max_filesize = 20M max_execution_time = 300 ***Very important can help reduce timeouts max_input_time = 300 output_buffering = 4096 disable_functions = exec, passthru, system, proc_open, proc_get_status, proc_nice, proc_terminate, proc_close, curl_multi_exec, show_source, dl, shell, system_exec, virtual error_reporting = E_COMPILE_ERROR E_ERROR E_CORE_ERROR Example:
MySQL Magento database optimization An optimised well performing Magento database is vital for overall system performance and speed. 1. Go to phpmyadmin 2. Choose your magento database 3. Select checkboxes for all tables 4. Choose Repair on select menu 5. Select with checkboxes all tables 6. Choose Optimize on select menu 7. Your database is ready It is always recommended to backup your database before performing any optimisation.
MySql Server Optimization (Advanced) 1. If you have subtable RAM (no less 8GB) - you can disable the swap file in the OS. This can be done via SSH command line >swap off 2. Update key_buffer_size MyISAM engine. go to mysql with superuser permissions (usually - root account on Mysql) enter two commands mysql > SHOW VARIABLES LIKE '%key_buffer%'; mysql > SET GLOBAL key_buffer_size = 536870912; 3. Changes of configuration file. Go to my.cnf (/etc or /etc/mysql directory) open file and create section [mysqld] [mysqld] key_buffer = 512M max_allowed_packet = 64M thread_stack = 192K thread_cache_size = 32 table_cache = 512 query_cache_type = 1 query_cache_size = 52428800 tmp_table_size = 128M expire_logs_days = 10 max_binlog_size = 100M sort_buffer_size = 4M read_buffer_size = 4M read_rnd_buffer_size = 2M myisam_sort_buffer_size = 64M wait_timeout = 300 max_connections = 400 4. Restart Mysql
Magento Optimization 1. Turn on Cache Control on Magento admin Admin Panel -> System -> Cache Management Enable each of Caches 2. Index database data Admin Panel -> System -> Index Management If you see "Update Required" - check index on left and "reindex data" on top right corner. Caching and indexing can be automated via a cron job to run at appropriate intervals. 3. Merge JavaScript files Admin Panel -> System -> Configuration on this page -> left column -> Developer (bottom of column) -> JavaScript Settings > Choose "Yes" and "Save config" Caution: Please test your store after enabling this option as incorrectly written or placed js code may not work when merged. 4. Merge CSS Files Admin Panel -> System -> Configuration on this page -> left column -> Developer (bottom of column) -> CSS Settings Choose "Yes" and "Save config" Caution: Please test your store after enabling this option. 5. Turn off debug Admin Panel -> System -> Configuration on this page -> left column -> Developer (bottom of column) -> Debug Choose "No" on "Profiler" row and "Save config" 6. Turn off logs Admin Panel -> System -> Configuration on this page -> left column -> Developer (bottom of column) -> Log Settings Choose "No" on "Enabled" row and "Save config" 7. Attributes optimization Admin Panel -> Catalog -> Attributes -> Manage Attributes enter 'Yes' only for used attributes. For non-used attributes use 'No'.
8. Run Compilation Admin Panel -> System > Tools -> Compilation and click on Run Compilation Process (top right corner) 10. Delete or uninstall non-used extensions/modules Example Admin Panel -> System -> Configuration -> Advanced -> Disable Modules Output and turn off Mage_poll (as example). 11. Delete non used Java-scripts from Theme page.xml file 12. Use custom Jquery scripts on page footer. Using of APC / Memcached as cache backend (Advanced) 1. Magento uses cache system on files, its possible to change this so that Magento uses to an APC or Memcached system. Installation, APC (root permissions) apt-get install php5-apc or pecl install APC or contact your hosting provider Add cache block to local.xml file <?xml version="1.0"?> <config> <global> <install> <date><![cdata[sat, 26 Jun 2010 11:55:18 +0000]]></date> </install> <cache> <backend>apc</backend> <prefix>alphanumeric</prefix> </cache> <crypt> <key><![cdata[870f60e1ba58fd34dbf730bfa8c9c152]]></key> </crypt> <disable_local_modules>false</disable_local_modules> <resources> <db> <table_prefix><![cdata[]]></table_prefix> </db> <default_setup> <connection> <host><![cdata[localhost]]></host> <username><![cdata[root]]></username>
<password><![cdata[f]]></password> <dbname><![cdata[magento]]></dbname> <active>1</active> </connection> </default_setup> </resources> <session_save><![cdata[files]]></session_save> </global> <admin> <routers> <adminhtml> <args> <frontname><![cdata[backend]]></frontname> </args> </adminhtml> </routers> </admin> </config> Installation, Memcached (root permissions) apt-get install php5-memcached or contact with your hosting provider Add cache block to local.xml file <?xml version="1.0"?> <config> <global> <install> <date><![cdata[sat, 26 Jun 2010 11:55:18 +0000]]></date> </install> <crypt> <key><![cdata[870f60e1ba58fd34dbf730bfa8c9c152]]></key> </crypt> <disable_local_modules>false</disable_local_modules> <resources> <db> <table_prefix><![cdata[]]></table_prefix> </db> <default_setup> <connection> <host><![cdata[localhost]]></host> <username><![cdata[root]]></username> <password><![cdata[f]]></password> <dbname><![cdata[magento]]></dbname> <active>1</active> </connection> </default_setup> </resources>
<session_save><![cdata[files]]></session_save> [b]<cache> <backend>memcached</backend> apc / memcached / xcache / empty=file <slow_backend>file</slow_backend> database / file (default) - used for 2 levels cache setup, necessary for all shared memory storages <memcached> memcached cache backend related config <servers> any number of server nodes can be included <server> <host><![cdata[127.0.0.1]]></host> <port><![cdata[11211]]></port> <persistent><![cdata[1]]></persistent> <weight><![cdata[2]]></weight> <timeout><![cdata[10]]></timeout> <retry_interval><![cdata[10]]></retry_interval> <status><![cdata[1]]></status> </server> </servers> <compression><![cdata[0]]></compression> <cache_dir><![cdata[]]></cache_dir> <hashed_directory_level><![cdata[]]> </hashed_directory_level> <hashed_directory_umask><![cdata[]]> </hashed_directory_umask> <file_name_prefix><![cdata[]]></file_name_prefix> </memcached> </cache>[/b] </global> <admin> <routers> <adminhtml> <args> <frontname><![cdata[backend]]></frontname> </args> </adminhtml> </routers> </admin> </config> Save local.xml file. Clear directory var/cache/. Restart Magento from Admin Panel. Mount var/cache TMPFS (command line, root permissions) mount tmpfs /path/to/your/magento/var/cache -t tmpfs -o size=64m