EPAM Systems EPAM White Paper Version 2.0: August 10 2012
Excellence in Software
Content 1. Introduction... 4 2. Business Case... 5 3. Problem Statement... 6 4. Proposed Solutions and Implementation... 8 4.1. Database layer... 8 4.2. Application layer... 10 4.3. Caches scaling... 10 4.4. Session transfer... 11 4.5. Data storage... 11 4.6. Web front-end... 12 5. Results... 13 6. Appendices... 15 Appendix A Authors... 15 Appendix B References... 15 Table of Figures... 16 Excellence in Software
1. Introduction Established in 1993, EPAM Systems, Inc. (NYSE: EPAM) is a leading global IT services provider with delivery centers throughout Central and Eastern Europe. Headquartered in the United States, EPAM employs over 7,300 IT professionals and provides services to clients worldwide using a global delivery model through its client management and delivery operations in the United States, Belarus, Hungary, Russia, Ukraine, UK, Germany, Kazakhstan, Sweden, Switzerland, Poland and Canada. EPAM s years of building complex engineering solutions have led us to develop deep expertise in a number of horizontal technology areas. These are now formalized as technological competency centers that enable our teams to develop, integrate and support best of breed solutions and practices. One such center is dedicated to Cloud Computing. EPAM Cloud Computing Competency Center is a team of experts providing professional support at every stage of our clients cloud projects from consulting and cost analysis to development and deployment. In addition to dozens of successfully completed cloud projects, we have built a portfolio of cloud offerings to suit the needs of EPAM s global development team and its clients including EPAM Private and Community Cloud and EPAM Cloud Orchestrator. This whitepaper illustrates EPAM s approach to make an existing application ready to be hosted in one of many public IaaS offerings. Several aspects of our approach are often being left out in a typical cloud migration project. As a result migration projects are delayed or fail due to unsatisfactory performance and stability once in the cloud. Through the example of a standard J2SE application, we will show you the whole process from definition of bottlenecks through their elimination and up to assessment of the results. 4 Excellence in Software
2. Business Case In the last several years we have witnessed Cloud Computing becoming a major trend in the IT industry. Based on recent research few organizations do not use of consider to use the cloud in near future. Benefits of strategic cloud adoption and correct cloud implementation are numerous and well known. And the end result in such case is well worth the effort. Organizations that adopt cloud strategically understand that their processes and skill-sets have to be revised accordingly. Organizations that adopt the cloud because it is trendy need that transformation as well, but often do not recognize that need until much later in the transition process. The challenge extends beyond processes and skills. Few existing applications are ready for the cloud. Having decided to move to the cloud, organizations often take their existing applications, deploy them in the cloud, as they would do it in-house and anticipate all the benefits of the cloud. The reality is that applications not designed or optimized for cloud will not be able to leverage much of its benefits. Cloud in such case is nothing more than virtual hosting. And as such applications will be constrained to the individual virtual machines rather than taking advantage of the scalability and availability of the entire cloud infrastructure. Furthermore, it is quite common that applications designed without scalability in mind once deployed into cloud environment exhibit significant performance degradation. The most common root cause is that an application is stripped of dedicated and managed hardware and placed into a shared environment. Another common reason is that the application does not fit well into the standardized virtual instance shapes in the cloud. There a several ways to solve this problem. The most obvious one is running the application on a more powerful instance i.e. vertical scaling. While solving the problem to some extent, this virtually nullifies some of the main cloud advantages, as vertical scaling is limited by performance of the most powerful instance and is by any means not the cheapest option. All of the aforesaid allows us to draw the following conclusion in order to make the best use of the cloud an application has to be scalable, in other words cloud-ready. Below we will showcase how to make our application scalable, while minimizing or avoiding amendments to its code. Clearly we could have just redesigned the whole thing from ground up, but that would have taken a lot longer, would have required significant attention from application business owners and could have introduced regression issues. Excellence in Software 5
3. Problem Statement As already mentioned above, this paper describes the approach we took to prepare an application for deployment in the cloud ( Cloudification ), based on an actual use case. As the target for our exercise we have chosen a content management system (CMS) that has been available for quite a while. It has never been designed with scalability in mind and was intended to be deployed on a single server. From a technical perspective the solution is a standard Java (J2SE) application built on top of the Hibernate library. It operates under a common application stack of Apache HTTP Server, Tomcat Application Server, and MySQL Database Engine that comprise a content management system. Figure 1 Bird s-eye Architecture Overview The system is fairly large and feature-rich. However its lengthy development has been reflected in legacy architecture. Our first step has been migrating the system to the Cloud as is. We deployed an absolute replica of the production environment in the Cloud, similarly to what organizations usually do. But for us it was just the first step to application Cloudification. So we got the whole system running on a single instance. Predictably application performance decreased. The cloud-friendly way to address this problem is to employ one of the defining cloud features elasticity. In other words, we were to begin scaling. By scaling we mean an ability to run an indefinite number of instance copies hosting the application and balancing the load. Clearly CMS has to remain functioning properly the requested content must be delivered, and relevant information received. And all of that must be consistent. Changes introduced to one instance must be distributed to all others. Excellence in Software 6
Addressing these problems directly required comprehensive architecture analysis to identify problem areas. Our approach was first to break the solution architecture into logical layers and subsystems, then to carry out the analysis of each layers capacity for horizontal scaling individually, and lastly evaluate possible issues with pulling horizontally scaled layers to operate together. Figure 2 - The System Broken into Logical Layers 7 Excellence in Software
4. Proposed Solutions and Implementation 4.1. Database layer As usual when dealing with some kind of a content management system, the first bottleneck here is the database. Horizontal scaling of SQL databases is notoriously difficult both practically and theoretically (Brewer's CAP theorem). Potential threats when scaling MySQL are: - Project code rewriting to support certain scaling approaches, which might lead to introduction of new bugs - Certain scaling methods may result in write bottlenecks - Some scaling methods are prone to synchronization issues - Scaling should not drastically increase fault probability - Database conflicts when multiple application instances update same or connected pieces of data Generic approaches to scaling MySQL horizontally are as follows: Master- Slave An unlimited number of slave instances can be added, but only one instance can be the master Great for applications with mostly READ operations Good for addressing I/O (network or disk) bottlenecks Good for geographic dispersion Requires application logic to separate read and write operations Replication is asynchronous, so instances are not guaranteed to be in sync Master instance can be scaled vertically 1) Master-slave replication the most common approach, involving splitting read/write operations. As read/write ratio can sometimes reach 90% to 10%, writing is performed in a single instance that has many replicas functioning in read-only mode. In our case this approach could not be applied as the application uses Hibernate. A wellknown flaw of this library is that we cannot be sure if the query is going to end in writing or not beforehand. Master- Master Robust. If one database is down, the application can continue functioning normally, with HAProxy routing all queries to the other master. Good for addressing I/O (network or disk) bottlenecks Good for geographic dispersion Requires stateful load balancing Need to continually monitor replication validity. Conflict resolution support in program code High complexity Excellence in Software 8
2) Master-master replication as it turns out we are unable to separate read and write operations, our next move is replicating instances, each one accepting both of them. The problem here is there can be data conflicts between instances when applying this approach to MySQL. Therefore, the application should be able to resolve them. In our case this approach requires substantially altering the code and thus cannot be applied. Sharding Good strategy for handling extreme database loads Easiest/most appropriate when most load is accessing a few tables and JOIN operations across shards are not required Success depends largely of sharding strategy and shard sizing May require painful periodic shard resizing Very high complexity Improves availability - The impact of a shard failure is small (affects only a portion of data), but sharding must be coupled with an additional strategy (like active-passive) to provide complete high-availability In terms of avoiding code refactoring, Sharding can arguably be considered an unbeatable approach to database scaling. The main difference from master-master replication is that none of the instances contains the whole amount of data, hosting a small part of it (a Shard) instead. The shards somewhat overlap to make the system failsafe. This approach is used in most of high-load systems, e.g. social networks and search engines. Sharding, however, requires adding a separate logical layer, directing read/write queries. Our options here is either adding it, which is a downright challenging task or use a 3 rd party application, serving as a single DB entry point and solving the tasks of breaking the DB on shards and assembling it later internally. We have found a suitable application called ScaleBase, however obtaining trial access to its functionality encountered unexpected problems forcing us to abandon this idea as well. Cluster Good solution for high-availability In-memory storage provides high-performance Good for scaling applications with heavy write loads Can scale specific portions of the cluster (query processing or storage) to target bottlenecks (CPU, RAM, I/O, storage volume) Synchronous replication between nodes makes cluster inappropriate for geographic dispersion Not a good fit for hosting environments with SAN Requires lots of RAM for storage nodes (roughly double the db size) High complexity 3) Building a MySQL Cluster was initially put aside due to its well-known complexity. However, it turned out to be the only approach satisfying our requirements. We were successful in building a scalable cluster, showing increase in performance when more nodes are added. However, building the cluster did pose several problems. The most notable ones required us to optimize database tables for cluster environment, namely: Truncate length of too long VARCHAR fields; Change table engine from InnoDB to NDB; Split complex indices, i.e. indices from multiple fields; Introduce new indices. Introducing these changes allowed us launching the database on a MySQL cluster. 9 Excellence in Software
4.2. Application layer This is the most difficult layer to scale if application is written without scalability in mind. This CMS application seemed to have a theoretical feasibility of horizontal scaling. It supported transactions, database operated at read committed isolation level and, generally, multiple instances could function in parallel, operating on the same data storage. At this point following caveats have been identified: - There were some concerns about valid transaction usage in application code. To address this concern we reviewed all the code, responsible for operation with the database, and fortunately enough, this proved our concerns wrong. - In-memory JVM caches which were used in application would have led to invalid instance states after data update happened on one of the instances. - Absence of session transfer functionality. 4.3. Caches scaling The application actively used caching techniques to speed up its performance. As a matter of fact, this was the most prominent issue identified throughout initial assessment, as these caches are completely unscalable. Caches are kept in-memory similarly to other structures in JVM s memory space. This implied that when scaling the application we will face the problem of cache synchronization, which in turn led to loss of instance state synchronization. Following chart demonstrates the problem: Figure 3 - Caches Scaling Cache was one of the most problematic areas of the project as the initial design of in-memory caches used by the solution did not allow horizontal scaling of application layer. In other words, we needed to refactor caching code to move from in-memory cache usage to embracing scalable solutions, such as memcached or Amazon ElastiCache. This has been done using Hazelcast. The cache functionality was implemented through separate isolated classes, making the refactoring quite easy, minimizing the risks and making the results easy to check. In the end both memcached and Amazon ElastiCache solutions turned out to be compatible and addressed our needs. 10 Excellence in Software
4.4. Session transfer Session transfer is the mechanism of moving sessions from on server to another in case of server failure. This allows addressing failures of a single server while a client is in the middle of a work session. Absence of such a mechanism leads to possible partial data or state information loss during server failures. The architecture analysis we undertook showed that the issue can be addressed utilizing built-in functionality of our application server Apache Tomcat. However, this brought additional complexity to the system and requires initiation of full code acceptance cycle, which was undesirable. Therefore, after assessing potential impact of such type of a failure (the application s architecture was mostly session-less), we came to the conclusion that we will not implement session transfer mechanism in this particular case. 4.5. Data storage The solution used file system storage for number of purposes: - storing cached html pages - storing downloadable site content Therefore scaling the application required the same content of file system storage to be available for all the application instances. So we had to implement a shared storage. One of the simplest and most reliable technologies to implement shared storage is the NFS protocol. We have introduced a separate NFS server that was mounted to all instances and acted as a single storage. Implementing such storage is usually associated with following issues: - Storage synchronization in case of replication - IO-bottle neck in case of shared storage - Scalability potential In our case performance of the NFS server turned out to be adequate and sufficient to the extent of tests performed by us. However, it is obvious that a single NFS-server is a bottleneck and a single point of failure. The simplest way to solve this problem is using cluster file systems, for example Gluster. This approach has been successfully implemented by us in a similar situation within another project, requiring NFS server scaling. 11 Excellence in Software
4.6. Web front-end To make active use of multi-instance horizontal scaling of this CMS system we needed a way to distribute incoming traffic across all instances. This could be achieved by setting up different kinds of load balancers. Our final choice was between Epam Cloud Load Balancer and Nginx The Load Balancing Service (LBS) of Epam Cloud is instrumental in delivering high-availability and scalability. The LBS handles each inbound request and decides which server is the best to process it. It also supports other enterprise-level requirements such as monitoring the web-farm state and providing support for sticky-session connections. Nginx is a modern, open-source, high-performance web server. It has advanced caching capabilities and can handle a considerable number of simultaneous client connections. This makes Nginx an excellent load balancer and reverse proxy for web services a single nginx server is able to distribute a large number of client connections to a number of upstream servers to handle the enduser requests. In practice both solutions could be used for our purpose. However, since advanced caching was crucial to our task, we have chosen Nginx as a load balancer for the system. 12 Excellence in Software
5. Results Having introduced the aforementioned changes, we moved on to comprehensive performance testing. This included running a considerable amount of tests supplying load on a single instance, then adding another instance and supplying the same load. The results showed significant reduction of response time as illustrated by charts below. Figure 4 - Mean Response Time Number of Instances Mean Response Time (ms) 1 3353 2 1560 3 1180 4 850 Excellence in Software 13
Figure 5 - Requests per Second Number of Instances Requests per second 1 15 2 7.3 3 6 4 5 14 Excellence in Software
6. Appendices Appendix A Authors 1. Ivan Pesin, Lead Maintenance Engineer, EPAM Systems ivan_pesin@epam.com 2. Aleksey Hariton, Senior Software Maintenance Engineer, EPAM Systems aleksey_hariton@epam.com 3. Iegor Sopov, Technical Writer, EPAM Systems iegor_sopov@epam.com Appendix B References 4. http://www.epam.com/ 5. http://nginx.org/ Excellence in Software 15
Table of Figures Figure 1 Bird s-eye Architecture Overview... 6 Figure 2 - The System Broken into Logical Layers... 7 Figure 3 - Caches Scaling... 10 Figure 4 - Mean Response Time... 13 Figure 5 - Requests per Second... 14 Excellence in Software 16
Global 41 University Drive Suite 202, Newtown (PA), 18940, USA Phone: +1-267-759-9000 Fax: +1-267-759-8989 EU Corvin Offices I. Futó st 47-53 Budapest, H-1082, Hungary Phone: +36-1-327-7400 Fax: +36-1-577-2384 CIS 9th Radialnaya Street, Building 2 Moscow, 115404, Russia Phone: +7-495-730-6360 Fax: +7-495-730-6361 1993-2012 EPAM Systems. All Rights Reserved.