Table of Contents. Overview... 1 Introduction... 2 Common Architectures Technical Challenges with Magento ChinaNetCloud's Experience...

Transcription

1

2 Table of Contents Overview... 1 Introduction... 2 Common Architectures... 3 Simple System... 3 Highly Available System... 4 Large Scale High-Performance System... 5 Technical Challenges with Magento... 6 PHP... 6 MySQL... 7 Customer-added code... 8 Admin / Indexing... 9 Search...10 Asset Sharing...10 ChinaNetCloud's Experience Architecture Selection Best Practice Configurations & Tuning Monitoring & Management Case Studies Clothing E-Retailer Flash Sales B2B Commerce Consumer E-Commerce Summary Getting Started New System Existing System Optimization About ChinaNetCloud Expertise Management Monitoring Solutions About Magento... 20

3 Overview Magento is the most common e-commerce platform on the Internet today, with over 150,000 retailers and websites using it. It's very powerful and flexible, very suitable for small and mid-range retailers building a serious on-line presence. Magento is the most common e-commerce platform on the Internet today, with over 150,000 retailers and websites using it. But Magento is a little complex and has many different deployment, architecture, and configuration options that directly affect its performance and reliability, especially as the traffic and/or product counts grow. In addition, the underlying hosting environment, increasingly cloudbased, also can greatly affect a Magento system. ChinaNetCloud has years of experience running and managing Magento systems of all sizes, up to 50,000 or more products, on public and private clouds, dedicated systems, and in highly-available architectures. As trusted server and infrastructure operations experts, numerous global and local retailers use ChinaNetCloud to design, build, and operate the underlying systems their Magento e-commerce shops run on. 1 ChinaNetCloud - Magento Server Management

4 Introduction E-commerce is one of the fastest-growing Internet applications in the world, especially in areas like China, Brazil, India, the Middle-East, and Southeast Asia. Driven by increasing middle-class purchasing power along with improved payment and delivery methods, these regions are joining the West in browsing and buying on-line. As the world moves to clouds a n d I a a S, m o r e a n d m o r e Magento systems are deployed o n A m a z o n AW S, A l i y u n, Rackspace, Slicehost, and more. Magento is probably the leading application platform leading this growth, powering over 150,000 of these small to medium-sized e-retailers all over the world. Its ease of use, flexibility, and customization options make it extremely attractive to companies seeking to grow up or out from typical hosted stores like Yahoo, Amazon, or Taobao/T-Mall. It's also increasingly seen as a replacement for small home-grown systems that rarely scale well with the business. Magento is a complex system with several major parts, different licenses, various architecture options for administration, search, caching, and high-availability. It also often needs to fit into a larger customer environment with membership, discounts, and global payment systems. In addition, infrastructure components such as databases, caching, CDNs, load balancers, and highavailability can all complicate Magento deployments and operations. Also, as the world moves to clouds and IaaS, more and more Magento systems are deployed on Amazon AWS, Aliyun, Rackspace, Slicehost, and more. While all powerful, flexible, and often cost-effective, these clouds always have limitations that can directly affect Magento performance and reliability. ChinaNetCloud - Magento Server Management 2

5 Common Architectures Below are three common architectures found in typical Magento systems, from basic simple systems up through fully distributed high-performance high-availability systems. Small Configuration <200 Concurrent Users Simple System The simplest real system uses one web server and one database server. This is suitable for dev/test environments, plus very small shops that are just getting started or very cost-sensitive. 3 ChinaNetCloud - Magento Server Management

6 Highly Available System A standard HA system has two of everything and adds a load balancing layer to split the traffic between the two web servers. This architecture is suitable for basic production installations, getting started with e-commerce, and for smaller systems. Medium Configuration Concurrent Users ChinaNetCloud - Magento Server Management 4

7 Large Scale High-Performance System For maximum performance, everything is split out and separated, including the Admin server and the search system, plus extra web servers and slave databases. And in front, a set of Varnish servers to help cache important components. Large Configuration >600 Concurrent Users 5 ChinaNetCloud - Magento Server Management

8 Technical Challenges with Magento Magento is a good and powerful system, is very flexible, easy to configure, and has thousands of extensions to add functionality. In addition, it's supported by a large network of developers and system integrators who typically configure, enhance, and setup Magento installations for their customers around the world. But like all systems, especially high-performance production ones, there are challenges and common problems the can harm the inexperienced Magento integrator or customer. PHP Magento is PHP-based, which is fine, as PHP powers most of the world's websites. But it's not always easy to deploy and manage at scale, especially as loads and connection counts increase. This is especially true when using Apache and modphp, which is the most common deployment method. This structure quickly runs out of RAM, CPU, or both as it scales, especially when placed on a public cloud with more limited resources. For example, on AWS with a medium instance, you get 1 CPU cores and about 4GB of RAM, and a large instance has 2 cores and 7.5GB of RAM. While these are nice cloud servers, they rapidly run out of CPU and RAM under load with Magento. In addition, PHP still has a reputation for security and stability problems, which take careful configuration, management, and monitoring to manage. These include secure configuration, APC or Zendopcode caching management, and more. ChinaNetCloud - Magento Server Management 6

9 Session management is another area where both PHP and Magento have issues, especially as the system scales and begins to involve multiple web servers. A lot of Magento developers like to put their sessions in the database, which is a major performance mistakes, as best practices dictate this should move to a cache like Memcache or Redis, especially on clouds or slower hardware. ChinaNetCloud has extensive experience with PHP systems of all types and sizes, including with Magento systems and their unique needs. MySQL Magento systems are MySQL-based, which is fortunate as MySQL is by far the most popular Internet application database in the world. However, it has challenges and requires careful configuration, management, and monitoring for best performance, security, and reliability. Magento often places a very heavy load on the database, especially when developers do not pay close attention to their SQL, custom database tables, and overall performance as the system scales. After all, any system runs well in development and when first used, but lots of Magento systems start to overload as they grow, due to database problems. For scale, Magento can only handle a single read-only slave, limiting its scalability on very large systems where multiple read slaves are common. A variety of solutions exist, mostly using a slave load balancer such as HAProxy. There are many other MySQL-specific issues such as MyISAM vs. InnoDB, high-availability configurations, tuning for performance, etc. 7 ChinaNetCloud - Magento Server Management

10 As one of the world's largest MySQL operators, ChinaNetCloud has many years of experience with largescale MySQL systems of all types and sizes, including many for Magento. These include customized bestpractice configurations, tuning, monitoring, and tools that can quickly find and fix performance and reliability problems on any site or system. ChinaNetCloud has many years of experience with large-scale MySQL systems of all types and sizes, including many for Magento. Customer-added code A major problem with most Magento systems is code added by the customer or system integrator, usually to add special features or to integrate into other systems. There are two common problems: Bad SQL and External HTTP Calls. The first, Bad SQL, is an obvious problem and mentioned above as a significant MySQL problem, where bad table design, poor indexing, or inefficient SQL causes severe performance problems the system fills with data and the load increases. For example, one ChinaNetCloud customer had very badly-performing SQL that had an IN clause with over 15,000 items in it. This query took minutes to run and was killing the system, but they couldn't see it or know this SQL as the main problem. Fortunately there are a number of advanced tools that can find these bad queries and help DBAs recommend improvements. In the above case, a small fix in the custom PHP code eliminated this problem and vastly improved overall system performance. The second problem, External HTTP Calls, is more difficult to deal with. Any real server has a fixed number ChinaNetCloud - Magento Server Management 8

11 of PHP processes it can run, usually on large heavily-loaded servers. But custom code often makes calls to external services such as Facebook or social systems. The easy solution is to add more processes, but this runs the risk of using up all the RAM and at high load, all the CPU in ways that also crash the site. In itself, this is not a bad thing, but it uses up one of the available PHP processes until that process / call completes. At high load and if that external process is slow for some reason, this can easily very quickly use up all the PHP processes, and lock up the system. The easy solution is to add more processes, but this runs the risk of using up all the RAM and at high load, all the CPU in ways that also crash the site. Admin / Indexing Magento includes a separate administration function which is often split out onto a separate PHP server on large systems so its load and processing doesn't affect the website and real users. In addition, Magento has a mostly static product catalog and search index which has to be rebuilt when new products are added. On larger systems with more than a few thousand products, this can take a long time and overload both the PHP server running the index process and the database server. These, in turn, have a very bad effect on system performance, in some cases for over an hour. If many products are added during the day, the result is a slow system all day. Best practices are to run the Admin system on a separate PHP server, though of course there is no way to separate out the database, which must be optimized and powerful enough to handle rapid re-indexing with minimal effect on the system and users. 9 ChinaNetCloud - Magento Server Management

12 Search Finding products is of course an important part of any e-commerce site, and as systems scale up the number of products, this can become a performance problem. This is especially true if the searches are done in the database (partly because they require MyISAM tables, when then complicate backups and index updates). Some larger systems use Solr as the search system, with much better performance and scale, plus it removes the core search functions from the main database, improving overall system performance. Some larger systems use Solr as the search system, with much better performance and scale, plus it removes the core search functions from the main database, improving overall system performance. Asset Sharing Another area that poses challenges Magento and all multi-web server setups is how to share assets such as product pictures, but also updated web components like lines, buttons, etc. With a single server, these are just put on that server, but with two or more, the problem arises of how to share and update those assets. The most common solution is probably NFS, but this can cause severe reliability problems in many configurations due to NFS instability and how it affects web servers. Even when perfectly stable, NFS creates a single point of failure (SPoF) that affects site reliability. Other solutions include rsync and other file sharing systems including complex cluster file systems such as GlusterFS. Each of these has advantages and disadvantages. ChinaNetCloud - Magento Server Management 10

13 ChinaNetCloud's Experience ChinaNetCloud has been running Magento systems of all sizes for several years, supporting e-tailers with up to 80,000 products, on dedicated hardware, private clouds, and public clouds. ChinaNetCloud supporting e-tailers with up to 80,000 products, on dedicated hardware, private clouds, and public clouds. Architecture Selection ChinaNetCloud runs hundreds of systems across all Internet verticals and industries, and have extensive experience in all types of architectures, from the simplest single server systems to large multi-hundred server, distributed systems with hundreds of millions of users. Such experience allows the development and recommendation of suitable architectures for every situation that provide an optimal mix of performance, reliability, flexibility, scale, and cost. Focused on simplicity, ease-of-maintenance, and flexibility, ChinaNetCloud-recommended architectures provide the best value for every customer. Best Practice Configurations & Tuning Sharing Once a system is designed and built, actual configuration and tuning of the system for real-world production conditions is very important. ChinaNetCloud has dedicated teams of senior engineers with deep experience in every common Internet technology, along with best-practice configurations for each component. Such configurations combine all the world's expertise and advice to guarantee best performance, reliability, 11 ChinaNetCloud - Magento Server Management

14 and security. Tuning the system is also an important aspect of operations, since every system is different in some way, especially for performance. Tuning the system is also an important aspect of operations, since every system is different in some way, especially for performance. This requires extensive monitoring, knowledge, tools, and procedures to extract the best possible performance, reliability, and security from each layer of the system, from hardware to the OS to the database to PHP/Java to the web servers and beyond. Monitoring & Management Every system needs to be managed, usually 7x24 for real-world production operations, and ChinaNetCloud is the largest world-class company running such systems across data centers, countries, industries, and technologies. This includes the world's most extensive monitoring system combined with sophisticated alert, log, and incident management to handle any problem. In addition, ChinaNetCloud uses an array of tools, training, systems, and processes to ensure rapid and best-practice handling of every request, project, problem or issue that can arise in system operations. This also includes providing advice on architectures, technologies, tools, data centers, hardware, CDNs, and every other aspect of large-scale Internet operations. ChinaNetCloud - Magento Server Management 12

15 Case Studies Clothing E-Retailer A clothing e-commerce site with over 30,000 products was having a number of performance and reliability problems with their Magento system, hosted on several high-performance physical servers. "Thanks for keeping our site up and fast during our fast growth." ChinaNetCloud migrated the system to a new data center, rebuilt the system, worked with the integrator on numerous configuration, tuning, and optimizations issues, and fixed additional CDN issues. - Clothing E-Retailer Customer Flash Sales A daily flash sales site with 80,000 products had many performance problems with Magento, with with 25+ very powerful servers. ChinaNetCloud worked extensively with the customer to improve performance at all levels, from hardware and I/O, to OS and DB tuning, to numerous SQL improvements to PHP, data center, and CDN upgrades. The results were faster performance, scaling to 10X traffic, to millions of users per day, on less than half the hardware, resulting in major improvements in traffic, sales results, and user experience. 13 ChinaNetCloud - Magento Server Management

16 B2B Commerce A large European B2B company launched a new Asia presence with a complex Magento system including full private cloud systems in both China and Japan, plus a central ERP interface gateway in Hong Kong to connect everything together via internal VPNs and European connections. " O u r e C o m m e r c e s i t e h a d to meet our strict security requirements, be scalable, and launch in a very short time. ChinaNetCloud s Private Cloud solution met all our requirements, and their 24x7 unlimited support makes my life easy." - Consumer E-Commerce Customer ChinaNetCloud worked with the integrators to design and deploy the clouds and has run the system successfully for several years, maintaining highavailability and performance, adjusting the underlying clouds and systems as needed for growth, business, and architecture changes. Consumer E-Commerce A large European retailer with numerous Chinese brickand-mortar stores launched a new on-line system based on Magento. ChinaNetCloud worked with the integrator to design, build, and manage the system. This included load testing and other preparations for large-scale traffic to handle promotions and other peak sales events. ChinaNetCloud - Magento Server Management 14

17 Summary Magento is a very good and extremely popular e-commerce platform, used globally by over 150,000 companies, and rapidly finding its way into developing markets such as China, India, SE Asia, and beyond. M a g e n t o r e q u i r e s c a r e f u l c o n f i g u r a t i o n, t u n i n g, monitoring, and management to provide the performance and reliability that modern e-commerce customers expect. While powerful and flexible, Magento requires careful configuration, tuning, monitoring, and management to provide the performance and reliability that modern e-commerce customers expect. ChinaNetCloud has that experience and has built and runs many Magento systems of all sizes. 15 ChinaNetCloud - Magento Server Management

18 Getting Started New System We will then work with you and your integrator to build, test, manage, and scale your system including our 7x24 management, monitoring, and optimization. Getting started on Magento is easy, either directly with ChinaNetCloud or through your system integrator. We can help design the most suitable architecture for your needs and budgets, and select the best hosting environment, on physical servers, a private cloud, or the various public clouds in each geography. We will then work with you and your integrator to build, test, manage, and scale your system including our 7x24 management, monitoring, and optimization. Existing System Optimization If you have an existing Magento system, we can do free audit checks and provide a variety of optimization and management services to ensure the system has the highest reliability, performance, scale, and security possible. Contact your ChinaNetCloud sales team today to learn more. ChinaNetCloud - Magento Server Management 16

19 About ChinaNetCloud Founded in Shanghai in 2008, ChinaNetCloud is the world's largest Internet Managed Services Company, focusing on server operations, especially Reliability, Performance, Scale, Security, and Cost Savings. F o c u s i n g o n l a r g e - s c a l e systems, ChinaNetCloud helps customers design, build, and operate any Internet-facing system and service, up to hundreds of millions of users and billions of page view per day. Focusing on large-scale systems, ChinaNetCloud helps customers design, build, and operate any Internet-facing system and service, up to hundreds of millions of users and billions of page view per day. Expertise As one of the world's largest Internet Server Management companies, ChinaNetCloud is the expert on large-scale Internet systems across many industries including e-commerce, mobile, gaming, advertising, video, and more. With hundreds of customers of all sizes, including extremely large systems with hundreds of millions of users, billions of transactions per day, and millions of queries per second, ChinaNetCloud knows how to design, build, and manage every type of system for maximum reliability, performance, scale, security, and cost savings. 17 ChinaNetCloud - Magento Server Management

20 Management ChinaNetCloud's core service is 7x24 management of large-scale Internet servers and infrastructure, from the data center to hardware to the OS to the database to the app server, cache, load balancer, firewalls, CDN and beyond. ChinaNetCloud system handles over 1,000 alerts and events per day, passing and managing them among various support, alert, escalation, and expert teams as needed. This world-class management operates as an extension of the customer's IT or operations team, handling all the issues, small and big, that are needed to keep large sites running well. These are handled by about a dozen functional teams, all using advanced tools and processes to provide best practice SLA-driven ITIL-inspired operations and support, 24 hours a day. Monitoring ChinaNetCloud's world-class monitoring system is one of the largest in the world, gathering over one billion data points per month, on average 150 per server, including deep OS, I/O, Database, and Application monitoring. In addition, other advanced monitoring systems are used at the network, syslog, security and other levels to carefully watch production systems for problems. With advanced classification, prioritization, and routing, the ChinaNetCloud system handles over 1,000 alerts and events per day, passing and managing them among various support, alert, escalation, and expert teams as needed. ChinaNetCloud - Magento Server Management 18

21 Solutions ChinaNetCloud has many pre-configured, ready-togo architectures and configurations for many popular applications and frameworks, including Magento e-commerce systems on physical servers, private clouds, and various public cloud providers such as AWS and Aliyun. ChinaNetCloud has many pre-configured, ready-to-go architectures and configurations for many popular applications and frameworks. More information about ChinaNetCloud is at 19 ChinaNetCloud - Magento Server Management

22 About Magento First released in 2008, Magento is one of the world's most popular E-Commerce platforms, used by over 150,000 retailers. Available at different price points, including the free community version to hosted versions to the Enterprise edition, it is extremely popular with e-tailers of all types and sizes. Today Magento is 100% owned by ebay. More information about Magento is at This Whitepaper provided by ChinaNetCloud for your information, based on our experience and public information we believe is accurate. No portion of this Whitepaper may be reproduced, copied, or distributed without written permission from ChinaNetCloud. Copyright 2013 by ChinaNetCloud - All Rights Reserved. ChinaNetCloud - Magento Server Management 20

23