Hybris E-Commerce Operations A White Paper by ChinaNetCloud
Table of Contents Overview... 1 Introduction... 2 Common Architectures... 3 Simple System... 3 Medium-sized Highly-Available System... 4 Large Scale High-Performance System... 5 Technical Challenges with Hybris... 6 Java... 6 MySQL... 7 Oracle... 8 Customer-added code... 8 Integration... 9 Asset Sharing... 9 ChinaNetCloud's Experience... 10 Architecture Selection... 10 Best Practice Configurations & Tuning... 10 Monitoring & Management... 10 Case Studies... 12 Clothing E-Retailer... 12 Flash Sales... 12 B2B Commerce... 12 Consumer E-Commerce... 13 Summary... 14 Getting Started... 15 New System... 15 Existing System Optimization... 15 About ChinaNetCloud... 16 Expertise... 16 Management... 16 Monitoring... 16 Solutions... 17 About Hybris... 18
Overview Hybris is one of the most common e-commerce platforms on the Internet today, with numerous larger retailers and websites using it. It is very powerful, flexible, and modular; while also very suitable for mid-range and larger retailers building a serious online presence. However, Hybris is somewhat complex and has many different deployments, architecture, and configuration options which directly effect its performance and reliability, especially as traffic and/or product counts grow. In addition, the underlying hosting environment, which is increasingly cloud-based, also can greatly effect a Hybris system. ChinaNetCloud has significant experience running and managing Hybris systems of all sizes, especially on private clouds in highly-available architectures. As trusted server and infrastructure operations experts, global and local retailers use ChinaNetCloud to design, build, and operate the underlying systems their Hybris e-commerce shops run on. ChinaNetCloud - Hybris Server Management Page 1
Introduction E-commerce is one of the fastest-growing Internet applications in the world, especially in areas like China, Brazil, India, the Middle-East, and Southeast Asia. Driven by the increasing middle-class purchasing power along with improved payment and delivery methods, these regions are joining the West in browsing and buying online. Hybris is one of the preeminent application platforms leading this growth at the higherend, powering many medium-sized or larger e-retailers all over the world. Its ease of use, flexibility, and customization options make it extremely attractive to companies seeking to grow up or out from typical hosted stores like Yahoo, Amazon, or Taobao/T-Mall. It is also increasingly seen as a replacement for large home-grown systems that rarely scale well with the business. Hybris is a complex system with several major parts, modules, and various architecture options for orders/oms, administration, search, products, integration, and highavailability. It also often needs to fit into a larger customer environment with memberships, discounts, and global payment systems. In addition, infrastructure components such as databases, caching, CDNs, load balancers, and high-availability can all complicate Hybris deployments and operations. Also, as the world moves to clouds and IaaS, more and more Hybris systems are deployed on Amazon AWS, Aliyun, Rackspace, Slicehost, and more. While all powerful, flexible, and often cost-effective, these clouds always have limitations that can directly affect Hybris performance and reliability. ChinaNetCloud - Hybris Server Management Page 2
Common Architectures Below are three common architectures found in typical Hybris systems, from basic simple systems up through fully distributed high-performance high-availability systems. Simple System The simplest real system uses two load balancers, two integrated web/application servers and two database servers. This is suitable for basic environments within small shops that are just getting started or are very sensitive towards cost. ChinaNetCloud - Hybris Server Management Page 3
Medium-sized Highly-Available System A standard HA system has double the networking equipment, splits out the web and application layers, and adds two additional application servers to handle additional load. This architecture is suitable for basic production installations, getting started with e- commerce, and for generally smaller systems. ChinaNetCloud - Hybris Server Management Page 4
Large Scale High-Performance System For maximum performance, everything is split out and separated, including the integration servers as well as extra slave databases. Located in front are a set of Varnish servers to help cache important components. ChinaNetCloud - Hybris Server Management Page 5
Technical Challenges with Hybris Hybris is a good and powerful system; it is very flexible, easy to configure, and has several modules and options based on specific business and operational needs. In addition, it is supported by a large network of developers and system integrators who typically configure, enhance, and setup Hybris installations for their customers around the world. Yet like all systems, especially large, high-performance production ones, there are challenges and common problems which can harm the inexperienced Hybris integrator or customer. Java Hybris is Java-based, which is fine, as Java powers most of the world's larger, more complex applications and websites. However it is not always easy to deploy and manage at scale, especially as loads and connection counts increase. Java is a good basic system, but it has unusual features including RAM management and monitoring challenges. On RAM management, a typical JVM allocates memory in an area called the heap, which is basically memory inside the JVM for use by the applications, especially Java objects. This RAM is harder to monitor than regular server RAM and when it runs low, very bad things happen, such as random failures and crashes among other strange things. The server will not swap or even let you know that this is happening. Java manages the heap via Garbage Collection, one of the best features of Java, yet also one of the hardest to manage and control in large-scale systems under heavy load. There are many different methods (five in Java version 7), configuration options, and schools of thought on good garbage collection, though most people just use the increasingly-better defaults. Meanwhile, a key challenge is that during major Garbage Collection (GC) operations, the entire application is frozen, a process that can last several seconds or longer and is hardly desirable for busy e-commerce applications. Some GC methods such as Concurrent Mark-and-Sweep (CMS) help minimize this at the expense of needing more RAM. Newer methods such as G1, introduced in JVM version 7, are also interesting to watch for future use. In this sense, heap monitoring and management can play a key part in Hybris' stability and scalability. This includes using servers, either physical or cloud/virtual, with plenty of RAM, usually at least 8GB and better with 16GB for smooth operations. ChinaNetCloud - Hybris Server Management Page 6
On the monitoring side, Java applications are also difficult to monitor, though Java has some standard monitoring tools that can be used if properly configured and secured. The most common is the Java Monitoring Extensions, JMX. JMX is a very nice system because it is built into Java and easily utilized by all infrastructure components like App Servers as well as applications like Hyrbis. However, it is important to realize that JMX is a Read-WRITE system, so it can both monitor and change applications and their behavior. Thus, it is critically important that proper readonly security is setup when using JMX. In terms of server and performance, Hybis and large Java applications in general need good hardware. For example, on AWS with a medium instance, you get 1 CPU core and about 4GB of RAM, with a large instance having 2 cores and 7.5GB of RAM. While these are nice cloud servers, they rapidly run out of CPU and RAM under load with Hybris. In this case, it is better to use extra-large four core and 16GB machines for heavy duty work. Session management is another area where both Java and Hybris can have issues, especially as most systems involve multiple web servers. Given that users can migrate between application servers in real-time or even failover situations, carefully handling session stickiness via tools like HAProxy or F5 cookies is important in providing a good customer experience. ChinaNetCloud has extensive experience with Java systems of all types and sizes, including with Hybris systems and their unique needs. MySQL Hybris systems are often MySQL-based, which is fortunate as MySQL is by far the most popular Internet application database in the world. However, it has its own challenges and requires careful configuration, management, and monitoring for the best performance, security, and reliability. Hybris often places a heavy load on the database, especially when developers do not pay close attention to their custom SQL, extra database tables, and overall performance as the system scales. After all, any system runs well in development and when first used, but lots of Hybris systems start to overload as they grow, primarily due to database problems. There are many MySQL-specific issues such as MyISAM vs. InnoDB, high-availability configurations, tuning for performance, slow log, SQL reporting, and generally just bad MySQL. As one of the world's largest MySQL operators, ChinaNetCloud has many years of experience with large-scale MySQL systems of all types and sizes, including ones built for Hybris. These include customized best-practice configurations, tuning, monitoring, ChinaNetCloud - Hybris Server Management Page 7
and tools that can quickly find and fix performance and reliability problems on any site or system. Oracle Hybris systems also often use Oracle as their core database, as a more serious, reliable, and powerful DBMS. However, it is also more complex and much more expensive, especially when deployed in its powerful HA mode, Oracle RAC. Oracle RAC is a very good system, but it requires special hardware, training, and configurations to run correctly. It also takes a full week of expensive expert time to install and setup, since every version of every hardware, firmware, and software component must be verified at every step. Other than Oracle and RAC-specific issues, the other performance, tuning, and SQL related challenges are the same for Oracle as they are with MySQL. ChinaNetCloud has extensive Oracle experience, both in-house, and via specialized Oracle partners for things like RAC and related SANs. This includes standard monitoring, backups, configurations, security, and more. Customer-added code A major problem with many Hybris systems is code added by the customer or system integrator, this is generally done in order to add special features or to integrate Hybris into other systems. There are two common problems: Bad SQL and External HTTP Calls. The first, Bad SQL, is an obvious problem and is mentioned above as a significant MySQL problem, where bad table design, poor indexing, or inefficient SQL causes severe performance problems as the system fills with data and the load increases. For example, one ChinaNetCloud customer had very badly-performing SQL that had an IN clause with over 15,000 items in it. This query took minutes to run and was killing the system, yet they were not able to see it or even know that this SQL was the main problem. Fortunately there are a number of advanced tools that can find these bad queries and help DBAs recommend improvements. In the above case, a small fix in the custom Java code eliminated this problem and vastly improved overall system performance. The second problem, External HTTP Calls, is more difficult to deal with. Any real server has a fixed number of Java threats it can run, especially within a user connected thread. However, custom code often makes calls to external services such as Facebook, social systems, external inventory, ERP, or more. ChinaNetCloud - Hybris Server Management Page 8
In itself, this is not a bad thing, but it uses up one of the available Java listener threads until that process/call completes. In a high-load situation if that external process is slow for whatever reason, this can easily use up all of the Java listener threads, slowing down the system for other users. The easy solution is to add more processes, but this runs the risk of using up all of the available RAM, and at high load, all of the CPU in ways that also can crash the site. Integration Hybris systems often include integration with other systems, either co-located or internal, as well as external systems such as with logistics, ERP, accounting, inventory, warehouse, store POS, or other partners. These integrations are handled in a variety of ways, often with specialized integration servers such as MuleSoft. This often creates the need for a separate integration server, often in a complex highavailability structure, with additional management, monitoring, and security. For example, Mulesoft's management console is based on Apache Tomcat, whose recent securityrelated exploits have led to additional Hybris system protection problems. In addition, integration servers are often using a wide variety of interaction methods with far-flung global partners including file transfers, RESTful APIs, HTTP/HTTPS calls, sftp/ftp pushes or pulls, email sending, and every other method imaginable. Each of these needs separate management and monitoring in real-world large-scale operations. This includes backups, rollbacks, and atomic replays of interchange data, as needed to fix or refresh / replay or even test and troubleshoot various integration problems. Asset Sharing Another area that poses challenges for Hybris and all other multi-web server setups is how to share assets such as product pictures, but also updated web components like lines, buttons, etc. With a single server, these are just put on that server, but with two or more, the problem arises of how to share and update those assets. Not to mention related problems such as how to quickly and efficiently deliver those assets to users across various geographies, CDNs, mobile, and other channels. The most common solution is probably NFS, but this can cause severe reliability problems in many configurations due to NFS instability and how it affects web servers. Even when perfectly stable, NFS creates a single point of failure (SPoF) that affects site reliability. Other solutions include rsync and other file sharing systems including complex cluster file systems such as GlusterFS, plus shared cloud storage such as Amazon S3 or Aliyun OSS. Each of these has their own advantages and disadvantages. ChinaNetCloud - Hybris Server Management Page 9
ChinaNetCloud's Experience ChinaNetCloud has been running Hybris systems of various sizes for some time, supporting large-scale global e-tailers with on dedicated hardware and private clouds, with numerous integration partners, network architectures, along with complex operational and support processes. Architecture Selection ChinaNetCloud runs hundreds of systems across all Internet verticals and industries, and have extensive experience in all types of architectures, from the simplest single server systems to large multi-hundred server, distributed systems with hundreds of millions of users. Such experience allows the development and recommendation of suitable architectures for every situation that provide an optimal mix of performance, reliability, flexibility, scale, and cost. Focused on simplicity, ease-of-maintenance, and flexibility, ChinaNetCloud-recommended architectures provide the best value for every customer. Best Practice Configurations & Tuning Once a system is designed and built, actual configuration and tuning of the system for real-world production conditions is very important. ChinaNetCloud has dedicated teams of senior engineers with deep experience in every common Internet technology, along with best-practice configurations for each component. Such configurations combine all the world's expertise and advice to guarantee best performance, reliability, and security. Tuning the system is also an important aspect of operations, since every system is different in some way, especially for performance. This requires extensive monitoring, knowledge, tools, and procedures to extract the best possible performance, reliability, and security from each layer of the system, from hardware to the OS to the database to PHP/Java to the web servers and beyond. Monitoring & Management Every system needs to be managed, usually 7x24 for real-world production operations, and ChinaNetCloud is the largest world-class company running such systems across data centers, countries, industries, and technologies. This includes the world's most extensive monitoring system combined with sophisticated alert, log, and incident management to handle any problem. ChinaNetCloud - Hybris Server Management Page 10
In addition, ChinaNetCloud uses an array of tools, training, systems, and processes to ensure rapid and best-practice handling of every request, project, problem or issue that can arise in system operations. This also includes providing advice on architectures, technologies, tools, data centers, hardware, CDNs, and every other aspect of large-scale Internet operations. ChinaNetCloud - Hybris Server Management Page 11
Case Studies Clothing E-Retailer A global e-commerce site with operations in over 100 countries came to China to build a large-scale Hybris system with complex integration, partners, networking, monitoring, support, and other requirements. ChinaNetCloud built the entire system on a private dedicated cloud with full HA systems, redundant and private-link networks, a full test and performance environment, plus dev/test systems. ChinaNetCloud also choose the data center, CDN, hardware, and worked with all of the various partners to build and manage the system, handling a long list of challenges along the way. Flash Sales A daily flash sales site with 80,000 products had many performance problems with their e-commerce system which ran on 25+ very powerful servers. ChinaNetCloud worked extensively with the customer to improve performance at all levels, from hardware and I/O, to OS and DB tuning, to numerous SQL improvements, data center, and CDN upgrades. The results were faster performance, scaling to ten times the previous amount of traffic up to millions of users per day. All of this was done on less than half the hardware, resulting in major improvements in traffic, sales results, and with user experience. B2B Commerce A large European B2B company launched a new Asia presence with a complex e- commerce system which included full private cloud systems in both China and Japan. In addition, a central ERP interface gateway was setup in Hong Kong to connect everything together via internal VPNs and European connections. ChinaNetCloud worked with the integrators to design and deploy the clouds, successfully running the system for several years, maintaining high-availability and performance, adjusting the underlying clouds and systems as needed for growth, business, and architecture changes. ChinaNetCloud - Hybris Server Management Page 12
Consumer E-Commerce A large European retailer with numerous Chinese brick-and-mortar stores launched a new online system. ChinaNetCloud worked with the integrator to design, build, and manage the system. This included load testing and other preparations for large-scale traffic to handle promotions and other peak sales events. ChinaNetCloud - Hybris Server Management Page 13
Summary Hybris is a very good and popular e-commerce platform for larger e-commerce sites, used globally by thousands of companies, and rapidly finding its way into developing markets such as China, India, SE Asia, and beyond. While powerful and flexible, Hybris requires careful configuration, tuning, monitoring, and management to provide the performance and reliability that modern e-commerce customers expect. ChinaNetCloud has that experience and has built and run Hybris systems of every size. ChinaNetCloud - Hybris Server Management Page 14
Getting Started New System Getting started on Hybris is easy, either directly with ChinaNetCloud or through your system integrator. We can help design the most suitable architecture for your needs and budgets, and select the best hosting environment, on physical servers, a private cloud, or the various public clouds in each geography. We will then work with you and your integrator to build, test, manage, and scale your system including our 7x24 management, monitoring, and optimization. Existing System Optimization If you have an existing Hybris system, we can do free audit checks and provide a variety of optimization and management services to ensure the system has the highest reliability, performance, scale, and security possible. Contact your ChinaNetCloud sales team today to learn more. ChinaNetCloud - Hybris Server Management Page 15
About ChinaNetCloud Founded in Shanghai in 2008, ChinaNetCloud is the world's largest Internet Managed Services Company, focusing on server operations, especially Reliability, Performance, Scale, Security, and Cost Savings. Focusing on large-scale systems, ChinaNetCloud helps customers design, build, and operate any Internet-facing system and service, up to hundreds of millions of users and billions of page view per day. Expertise As one of the world's largest Internet Server Management companies, ChinaNetCloud is the expert on large-scale Internet systems across many industries including e-commerce, mobile, gaming, advertising, video, and more. With hundreds of customers of all sizes, including extremely large systems with hundreds of millions of users, billions of transactions per day, and millions of queries per second, ChinaNetCloud knows how to design, build, and manage every type of system for maximum reliability, performance, scale, security, and cost savings. Management ChinaNetCloud's core service is 7x24 management of large-scale Internet servers and infrastructure, from the data center to hardware to the OS to the database to the app server, cache, load balancer, firewalls, CDN and beyond. This world-class management operates as an extension of the customer's IT or operations team, handling all the issues, small and big, that are needed to keep large sites running well. These are handled by about a dozen functional teams, all using advanced tools and processes to provide best practice SLA-driven ITIL-inspired operations and support, 24 hours a day. Monitoring ChinaNetCloud's world-class monitoring system is one of the largest in the world, gathering over one billion data points per month, on average 150 per server, including deep OS, I/O, Database, and Application monitoring. ChinaNetCloud - Hybris Server Management Page 16
In addition, other advanced monitoring systems are used at the network, syslog, security and other levels to carefully watch production systems for problems. With advanced classification, prioritization, and routing, the ChinaNetCloud system handles over 1,000 alerts and events per day, passing and managing them among various support, alert, escalation, and expert teams as needed. Solutions ChinaNetCloud has many pre-configured, ready-to-go architectures and configurations for many popular applications and frameworks, including Hybris e-commerce systems on physical servers, private clouds, and various public cloud providers such as AWS and Aliyun. More information about ChinaNetCloud is at www.chinanetcloud.com ChinaNetCloud - Hybris Server Management Page 17
About Hybris Founded in 1997 in Switzerland, Hybris is one of the world's most popular large-scale E- Commerce platforms, with numerous global customers. Ranked as a Leader by both Gartner and Forrester, Hybris is increasingly popular as the fastest growing major e- commerce platform company. Today Hybris is owned by SAP. More information about Hybris is at www.hybris.com This Whitepaper provided by ChinaNetCloud for your information, based on our experience and public information we believe is accurate. No portion of this Whitepaper may be reproduced, copied, or distributed without written permission from ChinaNetCloud. Copyright 2014 by ChinaNetCloud - All Rights Reserved. ChinaNetCloud - Hybris Server Management Page 18
Global Headquarters X2 Space 1-601, 1238 Xietu Road Xuhui District Shanghai, China 200032 Beijing Office Lee World Business Building #305 57 Happiness Village Road Chaoyang District Beijing, China 100027 +86 (21) 6422-1946 Sales@ChinaNetCloud.com www.chinanetcloud.com ChinaNetCloud - Hybris Server Management Page 19