The purpose of this benchmark was to compare the performance of

Transcription

1 Ruby on Rals Database Benchmark: Clustrx and MySQL by Clayton Cole and Nel Harkns The purpose of ths benchmark was to compare the performance of Ruby on Rals usng MySQL, an open-source database commonly used by web developers, and Clustrx, a scale-out SQL database optmzed for bg-data applcatons. In partcular, the benchmark tested how database performance (measured as transactons per second, or TPS) was mpacted by ncreasng concurrency (measured by total threads). The benchmark also collected data for average executon tme over the same range of concurrences. Rather than to serve as a drect comparson, the goal of the benchmark was to look at the performance that a typcal developer would face n a hosted envronment usng a standard MySQL server and how that developer could extend her database s performance so that Rals scales.

2 Ruby on Rals, often referred to as Rals, s an open-source web framework runnng on the Ruby programmng language that s used to create webstes and web applcatons. It s a full-stack framework, provdng web developers wth the ablty to gather nformaton from the web server, talk to or query the database, and render templates out of the box. Rals uses a Model-Vew-Controller (MVC) archtecture to organze ts applcaton programmng. Rals abstracts the database usng ORM (object relatonal mappng), wrappng an object model around the database to hde t from developers vew. As a result, developers can code up and test ther applcatons wthout havng to consder where or how ther data s stored. Tradtonal Ruby on Rals Scalng Lmtatons Ruby on Rals, used n conjuncton wth a sngle MySQL server, s one of the quckest ways to create a professonal web-based applcaton and s used by many stes experencng sgnfcant traffc growth due to ncreased popularty. World-class hostng of Rals-based webservers allows a company to deploy an applcaton quckly and to add more webservers as needed. Ths means that the ablty to scale a modern Ruby on Rals applcaton rests solely upon the scalablty of the database. It was mportant to ensure that benchmark code s avalable to the publc. See the Addtonal In- Lscum formaton secton of ths document for detals on Real-World Benchmark to Demonstrate Rals Scalablty exceped or ths Ruby on Rals benchmark, an aucton webste was selected to serve as a real-world smulaton of an operatng Rals applcaton. The aucton webste s manly transactonal, but some analytcal queres are also present. Thrteen scenaros were coded n Rals to test out a varety of database operatons whle dong somethng that makes sense to a web user, such as uploadng a photo to one s profle, placng a bd on an aucton, or vewng the most recent comments on an aucton. Goals The prmary goal of the benchmark was to compare performance of a standard web-hosted MySQL server wth standard three-node and sxnode Clustrx clusters usng real-world workloads. To accurately acheve ths goal, t was necessary to use a benchmark that runs n the same way, and that provdes the same statstcs, for both MySQL and Clustrx. Rals s an opnonated framework wth clear vews on how nformaton should be structured. Even though t s possble to overrde how Rals operates, ths s dscouraged because dong so would not be the Rals way. Except for addng ndexes, the database schema created by Rals was left alone for ths benchmark. accessng the code. To elmnate other sources of nconsstency, t was also mportant to avod the complextes and latences assocated wth ncludng HTTP data transmsson as a step n load testng. To do ths, the Vew and Controller portons of the Rals framework were bypassed, and the benchmark concentrated only on the Rals Model (representng database nteracton). The normal Rals framework and ActveRecord were stll used to access the database, but queres were generated by local Rake processes rather than HTML-based requests from a user Clustrx, Inc. All Rght Reserved

3 Partners Tact Knowledge, a software engneerng organzaton formed n 2002 n San rancsco, desgned the benchmark (n collaboraton wth Clustrx) and also coded t. Tact Knowledge desgns and bulds e-commerce, CMS, socal meda, and moble solutons for ts clents, whch nclude leadng enterprses worldwde. Blue Box, a leadng provder of managed applcaton hostng for platform and nfrastructure solutons, hosted the testng envronment and suppled the MySQL server and clent servers for the benchmark. Blue Box, founded n 2003, provdes fully managed hostng solutons for customers of all szes and complextes, utlzng both physcal hardware and vrtual servers. Percona, the oldest and largest ndependent MySQL support and consultng company, valdated the MySQL server confguraton used n the benchmark as part of a performance audt. Benchmark Methodology THE BENCHMARK Tact Knowledge, wth nput from Clustrx, desgned a smple schema representng the database relatonshps that an aucton webste mght requre (g 1). Tact Knowledge mplemented that schema wthn the Rals framework, then crafted 13 scenaros to demonstrate several user ste nteracton events that would cause database operatons from smple INSERTs to multple UPDATEs nsde a transacton and JOINs. user_detals d INT user_d INT N brthday DATETIME pcture BLOB P user_phones d INT user_d INT N phone_type INT number VARCHAR(255) user_addresses d INT tags d INT name VARCHAR(255) P U users d INT logn VARCHAR(255) frst VARCHAR(255) last VARCHAR(255) P U user_d INT N address_type INT street VARCHAR(255) cty VARCHAR(255) state VARCHAR(255) zp INT emal VARCHAR(255) aucton_tags status INT created_at DATETIME d BIGINT tag_d INT N P updated_at DATETIME bds user_d INT aucton_d INT N N d INT user_d INT N aucton_d INT N amount DECIMAL tmestamp DATETIME P user_comments d BIGINT user_d INT aucton_d INT comment TEXT N N P created_at DATETIME updated_at DATETIME aucton_photos d INT aucton_d INT ttle VARCHAR(255) mage BLOB N P auctons user_d INT d INT ttle VARCHAR(255) start_date DATETIME N P close_date DATETIME descrpton TEXT gure 1: Smple Schema 2012 Clustrx, Inc. All Rght Reserved

4 The 13 scenaros are detaled below. A full explanaton of these scenaros can be found n Appendx A. Note that most of these scenaros nvolve multple transactons, so care should be taken not to conflate concepts of scenaros and transactons. The calculatons for translatng scenaro counts to transacton counts are descrbed n the Dscusson: Accountng for TPS secton of ths paper. The 13 scenaros for ths benchmark were: Create user Create user wth detal Create user wth two phones Pck random user and upload pcture Pck random user and her addresses and add two more addresses Pck random user and her addresses, update ts felds and update any address Pck random user, pck random aucton and place new bd Pck random user, pck random aucton and comment on t Pck random user, create comment on her own aucton Pck random user, and edt her random comment Pck random user, create new aucton wth AuctonPhoto Pck random user, pck random aucton and tag t wth three new tags Pck random aucton and select ts comments ORDER-ed BY created_at column DESC or example, Rals used code such as ths: # Pck random User, pck random Aucton and place new Bd def scenaro_7 user_d confg) user = User.fnd_by_d(user_d) aucton_d aucton = Aucton.fnd_by_d(aucton_d) Bd.create user: user, aucton: aucton, tmestamp: DateTme. now, amount: Random.rand( ) end When ths code was executed, Rals generated SQL statements that looked lke ths: SELECT `users`.* ROM `users` WHERE `users`.`d` = 833 LIMIT 1 SELECT `auctons`.* ROM `auctons` WHERE `auctons`.`d` = 7 LIMIT 1 BEGIN INSERT INTO `bds` (`amount`, `aucton_d`, `tmestamp`, `user_d`) VALUES (1.42, 7, :18:41, 833) COMMIT 2012 Clustrx, Inc. All Rght Reserved

5 gure 2: Test Setup TEST SETUP The benchmark tests were conducted at Blue Box s hostng faclty n Ashburn, Vrgna, usng the same equpment and servces that they provde to ther exstng customer base. A logcal dagram of the test setup n depcted n gure Clustrx, Inc. All Rght Reserved

6 Network traffc was exchanged over a sngle 48-port Csco swtch. Network connectons used ggabt Ethernet. The Clustrx system (suppled by Clustrx) used CLX 4110 nodes grouped nto three-node and sx-node clusters. The Clustrx VIP (software load balancer bult nto the product) was used for balancng connectons. Each CLX 4110 node has: 8 cores 48GB RAM 896GB SSD 600GB HDD The MySQL machne (a former producton machne suppled by Blue Box) was confgured as follows: 8 cores (two Intel Xeon 5450: 4 GHz, 12MB cache) 128GB RAM 1.6TB spnnng HDD space (12 x 300GB Seagate 15k RPM SAS) Hardware RAID 10 Runnng Scentfc Lnux 6.2 MySQL verson The dedcated physcal servers (also suppled by Blue Box) were confgured as follows: 32-core machne (supportng eght vrtual prvate servers) 32 total cores (two AMD Opteron 6272 Lscum 2.1 GHz) 64GB RAM 4 x 1TB 7200RPM Seagate SAS Runnng Scentfc Lnux 6.2 Runnng OpenVZ for Vrtualzaton exceped 24-core machne (supportng sx vrtual prvate servers) 24 total cores (two AMD Opteron GHz) 48GB RAM 4 x 1TB Seagate SAS 15k RPM Runnng Scentfc Lnux 6.2 Runnng OpenVZ for Vrtualzaton There were 63 vrtual prvate servers (VPSs) total; each VPS was confgured as follows: 4 cores 8GB RAM 100GB dsk space Runnng Scentfc Lnux 6.2 Runnng Ruby 1.9.3p125 Runnng Rals Runnng Python Served n ether a test master or a test slave capacty TEST SEQUENCE The benchmark proceeded as follows: ENVIRONMENT PREPARATION The test admnstrator bult a campagn of tests (each campagn usng the same setup, but wth ncreasng concurrences). A cluster of the approprate sze was formed [Clustrx only]. A verson of the ntal dataset was bult n the target system (~475GB). Data was balanced across all nodes [Clustrx only]. MASTER CLIENT Test parameters were read from the test database and used to buld Ruby confguraton fles. These parameters ncluded such nformaton as test tme, workload, number of threads, and number of Ruby nstances. The Ruby confguraton fles were SCP ed to each slave clent partcpatng n the test. The slave clents were verfed as beng ready. A coordnated start command was sent va non-blockng SSH to all partcpatng slave clents. SLAVE CLIENTS The slave clents receved the command to start. The desred number of Ruby nstances was nvoked on each partcpatng slave clent. Automaton code verfed that the correct number of Ruby nstances was started. The slave clents slept whle the benchmark projects executed Clustrx, Inc. All Rght Reserved

7 RUBY ON RAILS BENCHMARK or every benchmark nstance nvoked, the desred number of threads was started, and each thread connected to the target database system. Each thread began runnng the prescrbed workload durng a warm-up perod wth statstcs collecton turned off. After warm-up, statstcs collecton was turned on, and the test ran for a set tme (10 mnutes). Results of the test were saved locally as a JSON fle. SLAVE CLIENTS The slave clents woke up. Automaton code checked that the resultng Rake output fles for all Ruby nstances ndcated success and ensured that all resultng JSON fles were OK. MASTER CLIENT The master clent woke up. All JSON fles were SCP ed to a local drectory. The resultng JSON fles were aggregated and ths data was nserted nto the local test database. CALCULATE RESULTS The test admnstrator calculated the resultng TPS for each test n the campagn usng the methodology descrbed n the Dscusson secton of ths paper Dscusson BUILDING THE INITIAL DATASET A data populaton tool provded by Tact Knowledge was used to buld an ntal 581MB dataset seed wth 10,000 users. Ths dataset was dumped to dsk (usng mysqldump) to ensure that a consstent seed dump fle was used each tme the ntal dataset was bult. Python code was used to generate unque offsets for d ntegers and assocated foregn keys n a SQL fle that INSERT/SELECT ed the seed rows and stacked the seed dataset. The ntal dataset sze dscussed throughout ths document was grown to approxmately 475GB. CHOOSING THE SIZE O THE INITIAL DATASET To ensure that nether MySQL nor Clustrx had an unfar advantage n beng able to load the entre dataset nto memory, the sze of the ntal dataset was chosen to sgnfcantly exceed total system memory szes. The ntal dataset sze used was approxmately 475GB (850 ncrements x 581MB), larger than the MySQL nstance s memory (128GB) and larger than the sum of the memory n each of the Clustrx nodes (6 x 48GB = 288GB). ACHIEVING THE RIGHT REPRESENTATIVE WORK- LOAD A varety of workloads could be desgned by weghtng the 13 benchmark scenaros n dfferent proportons. Because workloads can vary from ste to ste, a balanced workload was chosen, gvng each of the 13 scenaros an equal chance of beng used durng testng. Ths resulted n a rato of 52% reads and 48% wrtes, smulatng a setup where a web cachng layer absorbs many of the frequently repeated queres. REPRESENTING THE SITUATION THAT A REAL WORLD DEVELOPER WOULD ACE One of the goals of the benchmark was to look at the performance that a typcal developer would face n a hosted envronment when makng use of a standard md-grade MySQL server and to determne how that developer could extend her database s performance so that Rals scales. A comparson of a sngle MySQL server usng spnnng hard dsks aganst a cluster of computers usng sold-state dsks s not a drect comparson; rather, t s a way to show how snglenstance performance can be extended and to help measure the magntude of that addtonal performance Clustrx, Inc. All Rght Reserved

8 LOGICALLY GROUPING TEST RUNS TO EASE ANALYSIS Test runs were grouped nto campagns, representng the same workload and workng database run at ncreasng levels of concurrency. Performance for tests wthn a sngle campagn was represented as a lne on charts showng TPS versus concurrency or average scenaro executon tme versus concurrency. DESIGNING TEST CAMPAIGNS TO MAXIMIZE EECTIVENESS O EACH MARGINAL RUBY IN- STANCE Maxmzng clent effectveness meant ensurng that Ruby nstances had suffcent CPU, memory, network access, etc., to effectvely query the target database. To keep from unevenly loadng a physcal clent server (the vrtual server hosts, or VSHs), Ruby nstances for a gven level of concurrency were allocated such that each margnal Ruby nstance was nvoked on a vrtual server (the test slave clents) on a dfferent physcal server and not just on a dfferent vrtual server on the same physcal server whenever possble. As for other aspects of the test setup, the cohort of vrtual clent servers used for a gven test was the same for both MySQL and for Clustrx. MYSQL CONIGURATION Industry-standard InnoDB was used as the MySQL storage engne, rather than usng My- Lscum ISAM. MySQL was nstalled on a fresh operatng the /etc/my.cnf fle shown n Appendx B. exceped system mage usng yum. It was confgured usng CLUSTRIX CLUSTER ORMATION The Clustrx cluster was formed by executng a format command on all nodes to reset them to ther ntal state. Then the web UI was used to add the approprate number of avalable nodes as well as to set the IP address for the cluster s VIP, the cluster s name, emal contacts, and tme zone. OPTIMIZING THE LOAD ROM CLIENTS, RUBY INSTANCES, AND THREADS After testng the mpact of usng more or fewer threads per Ruby nstance, more or fewer clents, and more or fewer Ruby nstances per clent, t was found that clent performance was maxmzed (n terms of TPS) wth four threads per Ruby clent, regardless of clent or Ruby nstance count. ACCOUNTING OR TPS The statstcs that Tact Knowledge bult nto the benchmark measure the number of scenaros executed per second, but standard database comparsons use transactons per second (TPS). It was possble to calculate transacton executon count by multplyng each scenaro s executon count by the number of transactons per scenaro and addng n the Records Not ound (RN) errors that occurred durng that test run (each the result of a smple SELECT that returned zero rows). TPS was then calculated by dvdng total transacton count by test tme expressed n seconds. Ths approach was verfed for each of the scenaros by turnng on full query loggng on a sngle node; creatng a test wth one thread runnng on one Ruby nstance on one clent; settng a workload that used only that sngle scenaro; settng the test to run for a short perod of tme (fve seconds); pontng the sngle clent at the sngle node rather than the Clustrx VIP; and countng every sngle query, as well as the SELECT statements that tred to grab rows that ddn t exst. Results and Observatons The results of the benchmark tests were captured n terms of transactons per second (TPS) and n average scenaro executon tme (n whole seconds) for each of the three database setups observed (MySQL, three-node Clustrx, and sx-node Clustrx) over a range of concurrences from 1 to 10,000+ threads Clustrx, Inc. All Rght Reserved

9 gure 3: Benchmark Performance PERORMANCE IN TERMS O TPS Two runs of the Ruby on Rals performance benchmark on each of MySQL, a three-node Clustrx cluster, and a sx-node Clustrx cluster produced the followng chart showng the number of transactons per second (TPS) measured over a range of concurrences (see Appendx C for detals). The runs used for ths chart are the same ones used n the followng secton dscussng average scenaro executon tme, see gure 3. MySQL maxes out around 5,000 or 6,000 TPS at a concurrency of 256 threads. It drops down to about 3,000 or 4,000 TPS as concurrency hts 1024, whch represents about a one-thrd decrease n performance. Beyond that, performance decays steadly as concurrency ncreases. Note also, that MySQL makes t to ~9,250 connectons but not all the way to ~10,250 connectons because t could not complete the test at that level of concurrency. At 256 threads whch was MySQL s peak performance the three-node Clustrx cluster has about a 2.5X performance advantage over MySQL. At 1024 threads, as MySQL has started to drop off, the three-node cluster s httng ts peak performance of around 30,000 TPS, an 8x performance advantage. Beyond that, performance drops off as concurrency ncreases. The three-node Clustrx cluster successfully completed testng at ~10,250 concurrency and hgher concurrency levels were not tested. The sx-node Clustrx cluster has 15 tmes the performance of MySQL at 1024 threads. It reaches a peak of ~55,000 TPS, then drops very gently down to ~45,000 TPS as concurrency rses to ~10,250 threads. Hgher concurrency levels were not tested on the sx-node Clustrx cluster Clustrx, Inc. All Rght Reserved

10 gure 4: Average Scenaro Executon Tme AVERAGE SCENARIO EXECUTION TIME Two runs of the Ruby on Rals performance benchmark on each of MySQL, a three-node Clustrx cluster, and a sx-node Clustrx cluster produced the followng chart showng the average scenaro executon tme (n whole seconds) measured over a range of concurrences (see Appendx D for detals). The runs used for ths chart are the same ones used n the precedng secton, see gure 4. As dscussed n the Benchmark secton of ths document, scenaros are thngs such as addng a user, placng a bd on an aucton, or uploadng a pcture. Executng a scenaro nvolves several transactons, not just one transacton. See Appendx A for detals on the SQL statements generated by the scenaros. At 256 threads, MySQL completes requests to execute scenaros n about 0.1 seconds. But along the way to 1024 threads, t starts takng up to three seconds to answer requests. Out at ~9250 threads, MySQL takes an average of 7 to 8 seconds to complete a scenaro. The three-node Clustrx cluster completes scenaro requests n ~0.06 seconds at 256 threads. At 1024 threads, t completes these same requests n an average of ~0.13 seconds. At ~9250 threads, t completes scenaros n an average of 4.75 seconds. The sx-node Clustrx cluster completes requests n ~0.06 at concurrences of both 256 and Out at ~9250 threads, t completes scenaros n an average of ~0.7 seconds Clustrx, Inc. All Rght Reserved

11 Conclusons To put the benchmark results n perspectve from the standpont of a real-world organzaton s Ruby on Rals webste: 64 threads mght represent the pont where a small and growng organzaton debuts ts servce wthout much fanfare. 256 threads mght represent the pont where that organzaton has establshed tself and has a steady user base threads would be where the organzaton has started to become more popular threads would be where the organzaton s handlng world-class levels of servce. At low concurrency levels of up to 64 threads, MySQL provdes comparable performance (n terms of TPS and average scenaro executon tme) to Clustrx. However, a notceable dfference starts to emerge even by 256 threads. Although the average scenaro executon tmes are smlar, Clustrx exhbts a 2.5X performance advantage. As users begn to really pay attenton to the webste at 1024 threads t becomes tougher for MySQL to meet ther demands. TPS drops by one-thrd for MySQL, whle Clustrx hts ts TPS peak. There s an 8X performance advantage for Clustrx at ths pont. Average scenaro executon tme at 1024 threads s ~0.1 seconds for ether the three-node or sxnode Clustrx cluster, whch s practcally unchanged from lower concurrency levels. MySQL, however, exhbts long delays n turnng around scenaros at ths pont. The mpact on users s that they wll lkely become confused or annoyed and that ther user experence wll break down f they are watng an average of 2.5 seconds for MySQL to complete ther requests. By the tme servce levels rse to the world-class range of ~9250 threads, the sx-node Clustrx cluster s stll provdng 45,000 to 50,000 TPS and answerng scenaro requests n ~0.7 seconds whle MySQL s generatng 2000 TPS and turnng around scenaros n 7 to 8 seconds. MySQL s reactng so slowly at ths pont that varous severe tmeouts wll lkely occur, possbly brngng down the revenue-producng actvtes of the ste. In concluson, then, Ruby on Rals s an excellent framework for buldng scalable web propertes. MySQL and Ruby on Rals work well together for ntal development and deployment, for operatng wth low levels of concurrency, and for small dataset szes but eventually a Rals scalng lmt s reached as the database hts the lmts of a sngle MySQL nstance. A Clustrx scale-out SQL database soluton allows DevOps and admns to scale ther Rals soluton beyond ths lmt whle also provdng superor performance, hgh avalablty, and nherent fault tolerance. Addtonal Informaton or addtonal nformaton about mplementng the Benchmark: Retreve the code used to mplement the benchmark ( Go to Runnng the Ruby on Rals Database Performance Benchmark ( clustrx.com/blog/bd/227383/runnng- the-ruby-on-rals-database-performance- Benchmark) for a blog post descrbng the steps taken to run the benchmark Clustrx, Inc. All Rght Reserved

12 Appendx A: Typcal SQL for the Thrteen Scenaros 2012 Clustrx, Inc. All Rght Reserved

13 2012 Clustrx, Inc. All Rght Reserved

14 2012 Clustrx, Inc. All Rght Reserved

15 2012 Clustrx, Inc. All Rght Reserved

16 Appendx B: Contents of /etc/my.cnf [mysqld] datadr=/srv/data/mysql_datadr/mysql max_connectons=65535 max_user_connectons=65535 skp-grant-tables skp-name-resolve nnodb_fle_per_table log-slow-queres = /var/log/mysql/slow.log log-queres-not-usng-ndexes long_query_tme = 1000 nnodb_buffer_pool_sze = 100G query_cache_sze=0g query_cache_type=0 nnodb_flush_log_at_trx_commt = 1 nnodb_flush_method = O_DIRECT nnodb_log_fle_sze = 1G nnodb_log_fles_n_group = 3 default-storage-engne=innodb tmpdr=/srv/data/mysql_datadr tmp_table_sze=2g max_heap_table_sze=2g query_cache_lmt=25m socket=/var/lb/mysql/mysql.sock user=mysql # Dsablng symbolc-lnks s recommended to prevent assorted securty rsks symbolc-lnks=0 [mysqld_safe] log-error=/var/log/mysqld.log pd-fle=/var/run/mysqld/mysqld.pd 2012 Clustrx, Inc. All Rght Reserved

17 Appendx C: Data for TPS vs. Concurrency Graph 2012 Clustrx, Inc. All Rght Reserved

18 Appendx D: Data for Average Executon Tme vs. Concurrency Graph about clustrx, nc. Clustrx s the scale-out SQL database for Bg Data applcatons. Clustrx provdes a radcally smple SQL database that enables applcatons to scale to unlmted users, transactons, and data, whle elmnatng database shardng and au- tomatng fault tolerance. More than 125 Clustrx nodes are n producton n MySQL applcatons around the globe, wth more than 500 bllon transactons per month runnng through Clustrx databases worldwde. Customers nclude Symantec, AOL, MakeMyTrp, Photobox, and Massve Meda. or questons regardng Clustrx, please contact: nfo@clustrx.com Toll ree: Internatonal: about blue box Blue Box, founded n 2003, s a leadng provder of managed applcaton hostng for platform and nfrastructure solutons. Blue Box s enterprse clents receve whte-glove 24/7 support through ndustry-leadng techncal mplementaton and management expertse. Wth ownershp and control of the nfrastructure and full data locaton transparency, Blue Box delvers comprehensve hostng solutons wth game-changng uptme to enterprses and applcatons of any sze. or questons regardng Blue Box, please contact: sales@bluebox.net Toll ree: Clustrx, Inc. All Rght Reserved Internatonal: