Björn Kraus PHOENIX MEDIA Session Aware Full Page Caching For Magento With Varnish ESI
Was ist Varnish? Reverse Proxy Speichert beliebige Inhalte (HTML, CSS, JS, JPG/GIF) Gültigkeit über TTL/maxage im HTTP Header Steuerbar Volle Kontrolle durch Varnish Configuration Language (VCL) 5-10k Requests pro Sekunde möglich
Request-Ablauf (Cache Miss) GET / HTTP 1.1 Host: meinshop.de GET / HTTP 1.1 Host: meinshop.de Client Varnish Magento HTTP 1.1 200 OK HTTP 1.1 200 Cache-Control: max-age=20 HTTP 1.1 200 Cache-Control: max-age=20
Request-Ablauf (Cache Hit) GET / HTTP 1.1 Host: meinshop.de Client Varnish Magento HTTP 1.1 200 OK HTTP 1.1 200 Cache-Control: max-age=20
Was ist ESI? Edge Side Includes oder ESI genannt, ist eine Auszeichnungssprache um Webseiten-Komponenten dynamisch zusammenzubauen und diese als fertige Seite auszuliefern. http://de.wikipedia.org/wiki/edge_side_includes <html> <body> <p>the time is: <esi:include src="/cgi-bin/date.cgi"/> at this very moment.</p> </body> </html>
Request-Ablauf mit ESI GET / HTTP 1.1 Host: meinshop.de Varnish Client ESI Processing GET /pagecache/esi/... HTTP 1.1 Magento HTTP 1.1 200 OK HTTP 1.1 200 Cache-Control: max-age=20 HTTP 1.1 200 Cache-Control: max-age=5
ESI Request-Ablauf (Cache Hit) GET / HTTP 1.1 Host: meinshop.de Varnish Client ESI Processing Magento HTTP 1.1 200 OK HTTP 1.1 200 Cache-Control: max-age=20
Bringing it all together... 1. Varnish Modul für Magento 2. Dynamische Blöcke ersetzen mit ESI-Tags <esi:include src= /pagecache/esi/mini-cart/ /> 3. ESI in VCL aktivieren set beresp.do_esi = true 4. Magento Controller für ESI-Block- Generierung
Demo http://magento.phoenix-medien.de
PageCache powered by Varnish Kostenfreie Community Version http://www.magentocommerce.com/magen to-connect/pagecache-powered-by- varnish.html Enterprise Version mit ESI-Support
MySQL-Replication
Torsten Walluhn Magento Development Lead David Voge Infrastructure Lead
Need help with Magento performance?
Largest Magento Gold Partner in Europe with certified developers More than 40 Magento projects in 5 years with Magento Project implementation, Magento hosting, support & maintanence Focus on performance, migrations, integrations, time-to-market and TCO Working with medium and large retailers, manufacturers and brands First expert to launch Varnish ESI, Varnish Ajax, Redis, AWS and MongoDB
When to use a MySQL replication?
Use case #1: Large product catalog
Use case #2: High availability Reduce the risk Reduce single point of failure Disaster recovery Multi-location replication Mirroring
Use case #3: Load balancing Distribute workload across multiple computers Increase reliability Increase reliability through redundancy
Use case #4: Backing up from a slave File system snapshots Backup in a very short time No high computing load No outages during backup
We used MySQL replication for > 400k products Peak > 1MIO PI/day > 300/orders/hour Automatically scaled Web MySQL Cache
We used MySQL replication for
How did we achieve that?
Solutions Keep MySQL queries per page impression low Magento customization with good quality PHP 5.3.x Mysqlnd Optimize daemons and systems Web Server PHP MySQL Kernel Network File system Failover and disaster recovery plan Measuring and improving infrastructure performance
What can be done?
What can be done? Data Sharding Failover Sessions Backups Multiple slaves Location redundancy
Data Sharding write Magento read MySQL Master read MySQL Slave - Catalog - MySQL Slave - Other - sync
Data Sharding <catalog_read> <connection> <host><![cdata[srv]]></host> <username><![cdata[username]]></username> <password><![cdata[password]]></password> <dbname><![cdata[data_db]]></dbname> <model>mysql4</model> <initstatements>set NAMES utf8</initstatements> <type>pdo_mysql</type> <active>1</active> <use></use> </connection> </catalog_read>
What can be done? Data Sharding Failover Sessions Backups Multiple slaves Location redundancy
Basic failover implementation write Magento read MySQL Master MySQL Slave sync
Basic failover implementation Magento read / write MySQL Master MySQL Master
What can be done? Data Sharding Failover Sessions Backups Multiple slaves Location redundancy
Separation of the sessions write Magento read read / write MySQL Master MySQL Slave MySQL - Session - sync
Separation of the sessions write Magento read write read MySQL Master MySQL Slave MySQL Master - Session - MySQL Slave - Session - sync sync
What can be done? Data Sharding Failover Sessions Backups Multiple slaves Location redundancy
Create backups periodically in a short time Magento read / write MySQL Master 15 minutes sync Backup Host MySQL Slave snapshot Storage
What can be done? Data Sharding Failover Sessions Backups Multiple slaves Location redundancy
Each web server make use of it's own MySQL server running Magento running Webserver A write write Webserver B read read MySQL Slave sync MySQL Master sync MySQL Slave
What can be done? Data Sharding Failover Sessions Backups Multiple slaves Location redundancy
Location redundancy Magento Location A Magento Location B - standby- Webserver Webserver MySQL Master MySQL Slave MySQL Slave MySQL Slave sync
Where did Magento need help?
Core Problem: Backend Problem It is not possible to store the configurable product. Magento - Fronend - Magento - Backend - Workaround Backend server with it's own local.xml. read MySQL Slave write sync read / write MySQL Master
Core Problem: Failover Problem MySQL failover not possible directly by Zend Magento read / write Workaround Hardware load balancer Software load balancer Magento reconfiguration after outages. Programming a Zend extension. MySQL Master MySQL Master
Core Problem: Separation of the sessions Problem Currently not supported by Zend No separate session connector write MySQL Master Magento read MySQL Slave read / write Workaround Zend adjustment CGI has submitted a patch to Magento Inc. sync MySQL - Session -
Core Problem: Separation of the sessions --- a/build/app/code/core/mage/core/model/mysql4/session.php +++ b/build/app/code/core/mage/core/model/mysql4/session.php @@ -73,8 +73,8 @@ class Mage_Core_Model_Mysql4_Session implements Zend_Session_SaveHandler_Interfa public function construct() { $this->_sessiontable = Mage::getSingleton('core/resource')->getTableName('core/session'); - $this->_read = Mage::getSingleton('core/resource')->getConnection('core_read'); - $this->_write = Mage::getSingleton('core/resource')->getConnection('core_write'); + $this->_read = Mage::getSingleton('core/resource')->getConnection('session_read'); + $this->_write = Mage::getSingleton('core/resource')->getConnection('session_write'); }
What to measure, watch and optimize?
What to measure, watch and optimize? Replication status Failover process Synchronization of the database R/W splitting and more...
Pro / Contra
There are more advantages Pro Contra
Do s and Don ts Software Infrastructure Do understand how Magento uses MySQL resources. understand MySQL replication. check your shop in a test environment. monitor the performance and environment. implement a disaster and recovery plan. replication. implement a disaster and Don t trust 3rd Magento extensions - they can make use of wrong MySQL resources. use a small database to figure out an optimal architecture. enable replication on live system without testing. mix too much scenarios like data sharding and each node has it's own slave.