VMware vcloud Architecture Toolkit High Performance Data with VMware vfabric GemFire Best Practices Guide

Transcription

1 VMware vclud Architecture Tlkit High Perfrmance Data with VMware vfabric GemFire Octber 2011

2 High Perfrmance Data with VMware vfabric GemFire This prduct is prtected by U.S. and internatinal cpyright and intellectual prperty laws. This prduct is cvered by ne r mre patents listed at VMware is a registered trademark r trademark f VMware, Inc. in the United States and/r ther jurisdictins. All ther marks and names mentined herein may be trademarks f their respective cmpanies. VMware, Inc 3401 Hillview Ave Pal Alt, CA Page 2 f 36

3 Cntents High Perfrmance Data with VMware vfabric GemFire 1. Intrductin Overview Purpse Target Audience Scpe References vfabric GemFire Architecture Overview vfabric GemFire Tplgies vfabric GemFire General Administratin Guide Overview Installatin Cnfiguratin Mnitring General Administratin and Trubleshting vfabric GemFire and Spring High Level Tuning Overview... Errr! Bkmark nt defined. 5.2 JVM Memry Segments JVM Tuning and Best Practices vfabric GemFire n VMware Best Practices Overview Latency Sensitive Applicatins Best Practices if Virtualized Memry Sizing f Virtual Machines Running vfabric GemFire vcpu sizing f Virtual Machines Running vfabric GemFire Page 3 f 36

4 High Perfrmance Data with VMware vfabric GemFire Page 4 f 36

5 1. Intrductin High Perfrmance Data with VMware vfabric GemFire 1.1 Overview This High Perfrmance Data with VMware vfabric GemFire prvides infrmatin abut best practices fr the deplyment f data fabric systems. The guide describes the best practices fr VMware vfabric GemFire data caching systems and its varius design cnstructs. The dcument captures fur main deplyment patterns cmmnly used t implement enterprise data requirements: vfabric GemFire deplyed as an enterprise data management system. vfabric GemFire deplyed as L2 cache. vfabric GemFire deplyed fr HTTP sessin management. vfabric GemFire deplyed as a faster mass data mver fr example, real-time reprting. 1.2 Purpse This guide prvides best practice guidelines fr deplying vfabric GemFire. The recmmendatins in this guide are nt specific t any particular set f hardware r t the size and scpe f any particular implementatin. The best practices in this dcument prvide guidance nly and d nt represent strict design requirements because enterprise data requirements can vary frm ne implementatin t anther. Hwever, the guidelines d frm a gd fundatin n which yu can build many f ur custmers have used these guidelines t successfully implement an enterprise data fiber fr their enterprise applicatins. 1.3 Target Audience This guide assumes a basic knwledge and understanding f vfabric GemFire, data management cncepts, and virtualizatin with VMware vsphere. Architectural staff can use this dcument t gain an understanding f hw the system wrks as a whle as they design and implement varius cmpnents. Engineers and administratrs can use this dcument as a catalg f technical capabilities. 1.4 Scpe This guide cvers the fllwing tpics: vfabric GemFire Architecture This sectin prvides a high level best practice architecture fr varius tplgies that are part f the high perfrmance data slutin space. vfabric GemFire Best Practices This sectin cvers varius best practices pertaining t setting up a data fabric in prductin, and GemFire n vsphere best practice cnsideratins. vfabric GemFire Mnitring and Trubleshting Primer There are times when yu have t trublesht a particular vfabric GemFire applicatin prblem. vfabric GemFire is equipped with several tls such as GemFire Tl fr Mnitring (GFMn), Visual Statistics Display (VSD), and vsphere esxtp utilities which are very infrmative when trubleshting. vfabric GemFire FAQ In this sectin, we answer sme frequently asked questins abut the varius data fabric deplyments. Page 5 f 36

6 1.5 References It is recmmended that yu becme familiar with the fllwing dcumentatin: High Perfrmance Data with VMware vfabric GemFire vfabric GemFire User s Guide: Enterprise Java Applicatins n VMware Practices fr Perfrmance Tuning f Latency-Sensitive Wrklads in vsphere VMs Page 6 f 36

7 2. vfabric GemFire Architecture 2.1 Overview High Perfrmance Data with VMware vfabric GemFire vfabric GemFire is an in-memry distributed data management platfrm that can be spread acrss many virtual machines, JVMs, and GemFire servers t manage applicatin bjects. Using dynamic replicatin and partitining it ffers the fllwing features in the platfrm: data durability, reliable event ntificatin, cntinuus querying, parallel executin, high thrughput, lw latency, high scalability, cntinuus availability, and WAN distributin. The fllwing figure shws GemFire as the middle data tier that rchestrates data delivery frm the backend datastres t the cnsuming applicatins. As demand frm cnsuming applicatins increases, the middle tier data layer expands t apprpriately meet demand. Fr further persistence, resiliency data can be written behind t a backup stre like a relatinal database fr archival purpses. GemFire als prvides full persistence durability using its wn native shared-nthing persistence mechanism. Figure 1. vfabric GemFire Architecture Best Practice BP 1 Cmmn Distributed Data Platfrm Descriptin When data delivery is required t be at the highest speed pssible, if millisecnds and micrsecnds matter, setting up vfabric GemFire as an enterprise data fabric system is the crrect apprach. By ding s and as shwn in Figure 1, yu intrduce a cmmn data delivery and cnsumptin layer in-memry fr all enterprise applicatins data needs. This allws yu t benefit frm the scalability, availability, and speed f executin features f vfabric GemFire. Page 7 f 36

8 2.2 vfabric GemFire Tplgies High Perfrmance Data with VMware vfabric GemFire There are three main setup tplgies fr vfabric GemFire: client/server, peer-t-peer, and multisite. Each f these tplgies can be used standalne r cmbined t frm an extended, full featured distributed data management system Client/Server Tplgy In a client/server tplgy there are tw tiers, a client tier and a server tier. In Figure 1, the client and server tiers are depicted. The client tier cmmunicates with the server tier t search fr r update data bjects frm the server tier. In the client tier, standalne client caches 1, 2, 3, and 4 cmmunicate directly with the server tier. Figure 2. vfabric GemFire Client/Server Tplgy Page 8 f 36

9 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 2 Client/Server Tplgy Descriptin Client/server tplgy is the mst cmmnly used fr enterprise class applicatins. The client sends individual peratins t the server t update cached data, t satisfy a lcal cache miss, r t run an ad hc query. The server streams cache update and cntinuus query events t the client based n client subscriptins. Fr advanced tuning and increased thrughput capacity, yu can distribute the lad f netwrk traffic fr yur client/server traffic thrugh a different adapter than the peer-t-peer traffic by setting a server bind address. These gemfire.prperty lines specify different nn-default addresses fr the member: bind-address= , server-bind-address= Use client/server tplgy when any f the fllwing are requirements f an enterprise class applicatin: Dynamic server discvery The GemFire server lcatr utility dynamically tracks server prcesses and directs clients t new servers, giving clients indirectin frm explicit server infrmatin. Clients need t knw nly hw t cnnect t the lcatr services. They d nt need t knw where servers are running r hw many servers are available at any time. Server grups Yu can assign yur servers t lgical grups that yur clients can refer t in their cnnectin cnfiguratins. Fr example, yu might use grups t manually partitin yur data, with ne grup f servers hsting ne set f data and anther hsting anther set. Or yu might use a grup t direct all database-centric traffic t the subset f servers that are directly cnnected t a backend database. Servers can belng t multiple grups. Yur clients need t specify nly the grup t use and are islated frm having t knw which servers belng t which grups. Server lad balancing The GemFire server lcatr tracks current lad infrmatin fr all servers, directing new client cnnectins t the servers with the least lad. GemFire prvides a default lad prbe fr yur servers, which yu can replace with yur wn custmized plug-in. Server cnnectin cnditining Client cnnectins can be cnfigured t transparently time ut and be replaced with new cnnectins which allws verall server use t be rebalanced after new servers are started. This helps speed cnditining in situatins such as adding servers r recvery frm server crashes and ther dwntime. Autmated data and query updates Yur clients can subscribe t events in the server. These events can include data updates and updates t results fr cntinuus queries that the client has registered with the server. The server uses subscriptin queues t send the updates asynchrnusly. Server failver and high availability When servers crash, the client cnnectins autmatically fail ver t the remaining servers. If the servers are sending autmated updates t the clients, the update requests als autmatically fail ver. Yu can cnfigure redundancy in yur server subscriptin queues s that the failver des nt interrupt the stream f events frm the server side. Page 9 f 36

10 High Perfrmance Data with VMware vfabric GemFire BP 3 Client/Server Cmmn Sizes Because vfabric GemFire is hrizntally scalable, scalability is limited nly by the hardware resurces available. Cnfigure vfabric GemFire data management ndes t hst apprximately 1TB f data. While there is n prduct limit as t the number f JVMs deplyed in a data management system, up t 32 JVMs have been implemented in varius prductin systems. Thusands f clients can cnnect back t the data management systems accessing data. Client scalability can be managed thrugh cnnectin pls. Run within a 64-bit JVM, and a heap size f up t 32GB. When using -XX:CmpressedOps, the 32GB heap space uses 32-bit pinter addressing which saves large amunts f memry as ppsed t using 64-bit pinter addressing. With this apprach yu can cntinue t run inside a 64-bit JVM and benefit frm larger heap sizes, but with cmpressed pinter addressing. Any heap size beynd 32GB uses 64-bit pinter addressing because the -XX:CmpressedOps ptimizatin is limited t 32GB. This is limitatin in the Java ptimizatin and is nt specific t vfabric GemFire as GemFire supprts larger than 32GB heap sizes. Nte Client lcal cache generally starts with zer client side data and is enabled nly when needed fr perfrmance ptimizatin Peer-t-Peer In this tplgy, tw r mre intercmmunicating vfabric GemFire servers frm a distributed system. The data is distributed accrding t the data regin s cnfiguratin redundancy rules. Figure 3. Peer-t-Peer vfabric GemFire Distributed System Page 10 f 36

11 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 4 Peer-t-Peer Multihmed Machines Descriptin If running n multihmed machines, yu can specify a nn-default netwrk adapter fr cmmunicatin. In nn-multicast peer-t-peer situatins, cmmunicatin uses the bindaddress prperty. This address must be the same fr all vfabric GemFire servers within the distributed system. BP 5 Peer-t-Peer Sckets Highly cncurrent/high-thrughput deplyments need cnserve-sckets set t false and then limit the NIO thread pl servicing clients if (and nly if) the number f peer-t-peer sckets/wrker threads increases t the pint where cntext switching verhead degrades perfrmance. Mre ndes in the peer-t-peer cluster imply mre cnnectin/wrker thread verhead, and thus a pssible reasn t lwer the per-server NIO pl size. Cnversely, mre pwerful hardware with mre available cres, and running n a mre pwerful underlying netwrk fabric implies the ability t increase the per-server NIO pl size. Fr peer-t-peer threads that d nt share sckets, yu can use the scket-lease-time t limit the time that a scket sits idle. When a scket that belngs t an individual thread remains unused fr this time perid, the system autmatically returns it t the pl. The next time the thread needs a scket, it retrieves ne frm the pl. Scket-buffer-size determines the buffer size. Buffers shuld be at least as large as the largest stred bjects and their keys, plus sme verhead fr message headers. The verhead varies depending n wh is sending and receiving, but 32KB shuld be sufficient. Larger scket buffers allw yur members t distribute data and events mre quickly, but they als take memry away frm ther requirements. Nte This prvides excellent perfrmance even fr small update sizes, while nt killing the ptential fr larger-sized chunking t ptimize bulk peratins putall()/getall()/queries and rebalancing/failver/failback. BP 6 Peer-t-Peer hsts File BP 7 Use Lcatrs in Managed Peer-t-Peer Envirnments Verify that every peer-t-peer hst has a hsts file entry fr itself and fr all ther hsts n the LAN. The hsts file entry frmat shuld fllw hst + dmain (fr example, "gemserver1" and "gemserver1.vmware.cm") are present fr each IP address entered in the hsts file. Lcatrs using TCP/IP Using this methd yu run GemFire lcatr prcesses that manage the authritative list f active peer-t-peer distributed system members. These lcatrs are peer lcatrs. A new member cnnects t ne f the lcatrs t retrieve the member list which it uses t jin the system. Lcatrs are highly recmmended fr prductin systems. Fr prductin envirnments, always use at least tw lcatrs n different hsts. Page 11 f 36

12 Nte High Perfrmance Data with VMware vfabric GemFire The client/server tplgy is the mst cmmnly used in enterprise applicatins. There are sme rare cases when perfrmance cnstraints are s strenuus that a single hp is all that can be affrded t meet SLAs. Peer-t-peer tplgy typically has ne netwrk hp between peers as ppsed t client/server tplgy where there are tw netwrk hps, if yu assume that there are at least tw redundant servers with which a client can cmmunicate. Hwever, when using peer-t-peer tplgy it is assumed that the rich features f a client/server tplgy, such as cntinuus querying, registratin f interest, and cnnectivity thrugh pled cnnectins are nt needed. These features are available nly with the client/server tplgy. Client/server tplgy is the mst cmmnly used as it is the mst feature rich Multisite Tplgy In the case f multisite tplgy, as shwn in Figure 4, there are tw sites each with a distributed system. Within each site, ne server member is nminated as the gateway t prvide data distributin between sites in case f a failure event, r fr ther enterprise data distributin requirements. The Site 1 and Site 2 tplgy can lcate bth sites within ne datacenter, r the sites can be distributed gegraphically at different datacenters if needed. Figure 4. vfabric GemFire Multisite Tplgy Page 12 f 36

13 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 8 Multisite Tplgy Descriptin Use multisite tplgy in distributed data systems that require a rbust failver mechanism at the applicatin data layer. Use the cnflatin feature when using a gateway hub s that nly the latest updates are passed ver t the remte site. With cnflatin, earlier entry updates in the queue are drpped in favr f updates sent later in the queue. This is prblematic fr applicatins that depend n seeing every update. Fr example, if any remte gateway has a CacheListener that needs t knw abut every state change, yu shuld disable cnflatin. T enable cnflatin, yu can set the batch-cnflatin attribute t true within the gateway-queue cache cnfiguratin element. In a multisite installatin using gateways, messages can back up in the gateway queues if the link between sites is nt tuned fr ptimum thrughput. If a receiving queue verflws because f inadequate buffer sizes, it can becme ut f sync with the sender and the receiver is unaware f the cnditin. The gateway s <gateway> scket-buffer-size attribute shuld match the gateway hub s <gateway-hub> scket-buffer-size attribute fr the hubs the gateway cnnects t. Fr example: <gateway-hub id="eu" prt="33333"> <gateway id="us" scket-buffer-size="42000"> </gateway> </gateway-hub> <gateway-endpint id="us-1" hst="ushst" prt="11111"/> <gateway-queue verflw-directry="verflw" maximum-queue-memry="50" batch-size="100" batch-time-interval="1000"/> <gateway-hub id="us" prt="11111" scket-buffer-size="42000"> <gateway id="eu"> </gateway> </gateway-hub> <gateway-endpint id="eu-1" hst="euhst" prt="33333"/> <gateway-queue verflw-directry="verflw" maximum-queue-memry="50" batch-size="100" batch-time-interval="1000"/> Avid verflwing t disk when pssible by adjusting the maximum-queue-memry attribute t accmmdate needed memry. Hwever, shuld yu wish t verflw t disk, yu can easily d s t prvide additinal data reliability. Fr prductin systems and higher availability, set enable-persistence t true fr the gateway-queue attribute. This causes the gateway queue t persist t the disk stre specified in disk-stre-name. Althugh fr ease f illustratin we shw tw sites, typically yu wuld implement n+1 sites t achieve fault tlerance. The multisite tplgy can als span a WAN with multiple sites in, fr example, New Yrk, Tky, and Lndn. Refer t Figure 5. Page 13 f 36

14 Nte High Perfrmance Data with VMware vfabric GemFire Gateway hubs and gateways cmmunicate thrugh TCP/IP sckets. The gateway hub listens at a specified address and prt fr gateway cmmunicatin frm remte sites. Gateways are cnfigured with endpint infrmatin matching the remte gateway hub specificatins. The gateway sends cnnectin requests t the gateway hubs t establish tw-way TCP cnnectins. Fr infrmatin n the multisite cnfiguratin, refer t the Cnfiguring Multisite Installatins sectin f the vfabric GemFire User s Guide ( In additin t the site-t-site cmmunicatin, each gateway hub is a member in its wn distributed system. Figure 5 shws three glbal sites in New Yrk, Lndn, and Tky. Each site has a primary gateway and a backup gateway. It is imprtant t inspect and tune the cnfiguratin parameters f the WAN gateways. Figure 5. vfabric GemFire in a Glbal Multisite Cnfiguratin Page 14 f 36

15 2.2.4 Using vfabric GemFire as Simple L2 Cache High Perfrmance Data with VMware vfabric GemFire In Figure 6 a client/server tplgy is used t cnfigure vfabric GemFire as a Hibernate L2 cache. This cnfiguratin has the added benefits f faster perfrmance with relative ease f cnfiguratin. This is installed as a Hibernate plug-in and therefre n cde change is required. It als keeps the query results as distributed cache bjects thus imprving perfrmance and availability. Figure 6. Using vfabric GemFire as Hibernate L2 Cache Page 15 f 36

16 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 9 Hibernate L2 Cache Descriptin Turn n L2 cache in the Hibernate cnfiguratin (hibernate.cfg.xml): <prperty name="hibernate.cache.use_secnd_level_cache">true</prperty> Set regin.factry_class t GemFireReginFactry (hibernate.cfg.xml versin 3.3+): <prperty name="hibernate.cache.regin.factry_class"> cm.gemstne.gemfire.mdules.hibernate.gemfirereginfactry </prperty> Set the cache usage mde t: Read nly Used when yu d nt plan t mdify the data already stred in persistent strage. Read write Used when yu plan t bth read frm and write t data. Nn-strict write A special read/write mde that has faster write perfrmance. Use this nly if n mre than ne client updates the data at a time. Transactinal Allws fr transactin based data access. The cache mde can be set either using anntatin r in the Hibernate mapping file: T set using the Hibernate mapping file entity_name.hbm.xml: <hibernate-mapping package="package"> <class name="entity_name"...> </class> <cache usage="read-write nnstrict-read-write readnly"/>... </hibernate-mapping> T set the mde anntatins: imprt rg.hibernate.anntatins.cache; @Cache(regin = 'REGION_NAME', usage = CacheCncurrencyStrategy.READ_ONLY READ_WRITE NONSTRICT_READ_WRIT E TRANSACTIONAL) public class MyClass implements Serializable { }... Page 16 f 36

17 2.2.5 Using vfabric GemFire as an HTTP Sessin Cache High Perfrmance Data with VMware vfabric GemFire Best Practice BP 10 Tplgies fr HTTP Sessin Management Descriptin Either client/server, peer-t-peer, r multisite vfabric GemFire tplgies can be used t achieve HTTP sessin replicatin. If dealing with user sessin data that must be cmpletely fault tlerant, use multisite vfabric GemFire tplgy and HTTP sessin management. Fllw the recmmended setup in the HTTP Sessin Management Mdule sectin f the vfabric GemFire User s Guide ( It is relatively straightfrward with minimal change t cnfigure HTTP sessin replicatin with GemFire n VMware vfabric tc Server. In Figure 7, vfabric GemFire is used fr HTTP sessin replicatin that can be easily achieved when plugged int vfabric tc Server. Figure 7. Using vfabric GemFire fr HTTP Sessin Replicatin Using vfabric GemFire as a Faster Data Mver Best Practice BP 11 Real-Time Reprts Descriptin vfabric GemFire client/server tplgy is mst suited fr real time reprt setup. This allws yu t mve rapidly changing data t the cnsuming end pint client cache t present the data in real time. vfabric GemFire features such as cntinuus querying and functin executin can help in the implementatin f a business critical real time reprts. Page 17 f 36

18 High Perfrmance Data with VMware vfabric GemFire 3. vfabric GemFire General Administratin Guide 3.1 Overview The fllwing sectins summarize sme high level best practices. There are additinal details in the Administratin sectin f the vfabric GemFire User s Guide. 3.2 Installatin T dwnlad vfabric GemFire, g t Fllw the installatin instructins at: Cnfiguratin The mst ntable cnfiguratin files within vfabric GemFire are gemfire.prperties, gemfirelicenses.zip, and cache.xml. gemfire.prperties Cntains the settings required t jin a distributed system. Cnfiguratin includes system member discvery, cmmunicatin parameters, security, lgging, and statistics. Fr a detailed descriptin f the parameters within this file, refer t the vfabric GemFire User s Guide. gemfirelicense.zip The license file which shuld never be unzipped. Nte This is the license file fr vfabric GemFire 6.5. With vfabric GemFire 6.6 and later, licensing is dne using serial numbers. Refer t the vfabric GemFire User s Guide fr details. cache.xml The declarative cache cnfiguratin file. This file cntains XML declaratins fr cache, regin, and regin entry cnfiguratin. It is als used t cnfigure disk stres, database lgin credentials, server and gateway lcatin infrmatin, scket cnfiguratin, and s frth. Page 18 f 36

19 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 12 Cnfiguratin Descriptin D nt t unzip the gemfirelicense.zip file leave it intact. Each f the three cnfiguratin files has a default name, a set f file search lcatins, and a system prperty that can be used t verride the defaults. T use the default file specificatin, place the file at the tp level f its directry r jar file. The system prperties are standard file specificatins that can have abslute r relative pathnames and filenames. If yu d nt specify an abslute file path and name, the search lks thrugh all the search lcatins fr the file. The gemfire.prperties file can be specified with the system-level Java prperty -DgemfirePrpertyFile=<valid file/path>. Yu can verride any GemFire prperty set in the file r by the CacheFactry API with a system-level Java argument that fllws the pattern -Dgemfire.<prperty-name>=<prperty-value>. Deply the same gemfirelicense.zip n all members f the peer-t-peer tplgy, fr vfabric GemFire 6.5. Fr vfabric GemFire 6.6 use the same license key n all members f the peer-tpeer tplgy. All peer-t-peer members f the distributed system must have the same versin f vfabric GemFire. Clients can be up t ne majr release behind. Fr example, any 6.x client interperates with any 6.x r 7.x server, but nt with an 8.x server. The vfabric GemFire prperty aut-start=true must be cnfigured fr the agents fr any versin f GemFire 6.5. Fr highly cncurrent wrklads, set the GemFire prperty cnservesckets=false n the data management ndes (DMNs). If the scale is large and t many sckets (and assciated threads t service thse sckets) are created between the DMNs, tune the CacheServer (a cnfiguratin element in the DMN cache.xml cnfiguratin) t reduce the NIO thread pl servicing client requests. This places a hard upper limit n the pssible number f DMN peer-t-peer cmmunicatin sckets. Refer t the vfabric GemFire User s Guide fr infrmatin: <cache-server max-cnnectins="<integer number>" </cache-server> max-threads="<integer number>"> Place the default files either in the current directry frm which yu start the GemFire server r n the CLASSPATH. If yu wish t change the default names f these cnfiguratin files, yu can set the fllwing prperties t verride them. Nte These prperties are useful t script the deplyment r mvement f the cde base frm Dev t QA, and then t prductin, where there might be a separate set f cnfiguratin files fr each envirnment. Depending n which envirnment is deplyed t yu can rtate the apprpriate files in the gemfireprpertyfile, gemfire.cache-xml-file, r gemfire.license-file. Set -Djava.net.preferIPv4Stack=true in the start script fr all servers, peers, and lcatrs. Page 19 f 36

20 3.4 Mnitring High Perfrmance Data with VMware vfabric GemFire Best Practice BP 13 vfabric GemFire Mnitring Tls Descriptin vfabric GemFire is a specialized prduct and it is imprtant that administratrs are familiar with the available mnitring tls. The vfabric GemFire Tls Guide ( details GFMn and Visual Statistics Display (VSD) tls available fr mnitring. The GFMn tl mnitrs a vfabric GemFire system in real time, prviding health infrmatin, detailed peratinal and cnfiguratin data, system alerts, thrughput perfrmance, and statistics fr system members and cnnected clients. The VSD tl reads GemFire statistics and prduces a graphical display fr analysis. Cnfigure the vfabric GemFire prperty statistics-enabled=true t generate statistics files that can be viewed with VSD. This can be critical t trublesht ptential prblem areas r help t diagnse a prblem. Yu can als use the VMware vfabric Hyperic GemFire plug-in that prvides a live data user interface fr viewing metrics in real time. Refer t the vfabric Hyperic Guide: Yu can als use esxtp t mnitr vsphere, refer t the trubleshting sectin f the Enterprise Java Applicatins n VMware General Administratin and Trubleshting Best Practice BP 14 General Administratin Descriptin Fr managing disk stres, security, system lgs, trubleshting, the Cmmand Line Utility, and fr administering the Distributed System, refer t the vfabric GemFire User s Guide. D nt cnfigure firewall restrictins between the LAN hsts. There must be least fur dedicated prts pen thrugh any intervening firewall (cnnecting t the peert-peer hsts): ne fr the lcatr prt, ne fr the server prt, ne fr the agent HTTP prt, and ne fr the agent RMI prt. If trubleshting a cnnectivity prblem, set lg-level=fine n all sides f the cnnectin. This shuld always include the lcatr, as it is the first pint f cntact fr cnnectivity. At a fine lg level, yu can immediately see whether a scket cnnectin is made, and if it is, why and where the cnnectin is rejected. After trubleshting it is imprtant t revert t the default lg level, r typically in prductin, t the lg-level=errr. When yu first begin t diagnse a ptential cnnectivity r general prblem with the system, start with telnet t test whether a remte lcatr, server, r agent prt is reachable. Page 20 f 36

21 High Perfrmance Data with VMware vfabric GemFire BP 15 General Trubleshting The vfabric GemFire User s Guide has a detailed sectin n Trubleshting and System Recvery. Fllw thse instructins. BP 16 Trubleshting SYN Ckies When trubleshting perfrmance prblems, check t see yu are nt impacted by SYN ckies. SYN ckies are the key element f a technique used t guard against SYN fld attacks. Daniel J. Bernstein, the technique's primary inventr, defines SYN ckies as particular chices f initial TCP sequence numbers by TCP servers. In particular, the use f SYN ckies allws a server t avid drpping cnnectins when the SYN queue fills up. Instead, the server behaves as if the SYN queue had been enlarged. The server sends back the apprpriate SYN+ACK respnse t the client but discards the SYN queue entry. If the server then receives a subsequent ACK respnse frm the client, the server is able t recnstruct the SYN queue entry using infrmatin encded in the TCP sequence number. T check fr the presence f SYN ckies: grep SYN /var/lg/messages Aug 2 12:19:06 w1-vfabric-g1 kernel: pssible SYN flding n prt Sending ckies. Aug 2 12:54:38 w1-vfabric-g1 kernel: pssible SYN flding n prt Sending ckies. Aug 3 10:46:38 w1-vfabric-g1 kernel: pssible SYN flding n prt Sending ckies. T determine whether r nt SYN ckies are enabled (1 is n, 0 is ff): $ cat /prc/sys/net/ipv4/tcp_synckies 1 T temprarily disable SYN ckies (changes at rebt): # ech 0 > /prc/sys/net/ipv4/tcp_synckies Permanently disable SYN ckies: Add\mdify the fllwing in /etc/sysctl.cnf # Cntrls the use f TCP synckies net.ipv4.tcp_synckies = 0 Page 21 f 36

22 4. vfabric GemFire and Spring High Perfrmance Data with VMware vfabric GemFire Best Practice BP 17 vfabric GemFire and Spring Descriptin Use Spring t cnfigure GemFire servers and regins rather than manually creating them with applicatin cde. This allws yu t centralize yur applicatin service cnfiguratin, as ppsed t having the Spring cntext cnfiguratin plus a separate cache.xml file. Yu can eliminate the need t implement Declarable n yur cache laders, cache listeners and cache writers. Spring takes care f binding these cmpnents int GemFire regins thrugh dependency injectin. These cmpnents can als be shared amng regins as singletns using Spring's nrmal DI techniques. Yu can leverage advanced techniques t cnfigure GemFire via SpEL, selectively expsed cnfiguratin parameters, and s n. Use the Spring GemFire prject t mve yur GemFire cnfiguratin int the Spring cntext mre easily. Refer t Spring GemFire Hme at Use the GemFire schema extensin t the Spring cntext cnfiguratin t simplify the cnfiguratin f the varius GemFire cmpnents further with validated cnfiguratin prperty names (Sectin 1.1 in the Spring GemFire dcumentatin). Use GemfireTemplate t simplify interactins with the GemFire APIs. GemfireTemplate includes many best practice techniques fr dealing with resurce management and multiple threads in a virtual machine wrking with GemFire. GemfireTemplate eliminates the need fr yu t deal with checked exceptins. The GemfireTemplate prvides the best practice technique fr ensuring thread safe access t GemFire resurces in a single virtual machine. GemfireTemplate prvides utility methds t access and manage data mre simply. Transactin management using Spring and Spring GemFire: This prvides a prtable, well integrated way t prvide transactins t yur applicatin t prmte well defined behavir fr multiple threads mdifying the data. Autmatically cnfigures the GemFire server with best practice recmmendatins fr safest use in a multithreaded envirnment by enabling cpyonread t prevent client threads frm inadvertently editing data cntents in a nn-transactinal way. Use InstantiatrFactryBean t autmatically generate an efficient Instantiatr: Reflectin is the default technique used t serialize and deserialize data acrss the entire distributed data management system. If data serializatin is a bttleneck in yur applicatin, the general best practice recmmendatin is t implement custm instantiatr lgic t speed up the serializatin prcess. The InstantiatrFactryBean takes a list f dmain types (and a unique integer ID fr each type t efficiently serialize type inf as an integer instead f a string) and autmatically generates instantiatrs using the ASM bytecde library that prevents having t use reflectin t serialize and deserialize bject data. Page 22 f 36

23 High Perfrmance Data with VMware vfabric GemFire With transactins, yu must use care if perating in an envirnment where yu might access different types f transactinal resurces. This is true even if using JMS and JDBC tgether in a single Spring applicatin and they must be crdinated int a single transactin. Spring gives yu patterns that yu can use t reslve this. One pattern is simply nt t crdinate at all. Yu define multiple transactin managers, and then chse the apprpriate transactin manager that shuld apply t a particular set f cde (either r prgrammatic transactin). In this way, the cde that updates GemFire directly wuld use the GemFire transactin manager, and the cde that wrks with sme ther resurce wuld use the apprpriate transactin manager fr that resurce. Anther pattern is t crdinate thse tw resurces int a single JTA transactin. This requires either a JTA transactin manager like JOTM r Atmiks, r yu wuld have t run yur applicatin in a Java EE cntainer. Yu culd then enlist all yur transactinal resurces int a single JTA transactin manager, and nt have t wrry abut crdinating them. The GemFire use f ptimistic transactins shuld help reduce the chance fr deadlcks. Als, using GemfireTemplate shuld help reduce this further. If using a write-thrugh apprach with a CacheWriter writing t a database r sme ther transactinal resurce, then it is highly recmmended t use a JTA transactin manager t direct that GemFire s updates and any ther resurce are autmatically part f the same transactin. With write-behind style updates f a database, this is nt as much f an issue because the write t the database is ccurring in a different thread frm the cache updates. Write-behind is typically mre cmmn due t the perfrmance increase it prvides. Page 23 f 36

24 5. High Level Tuning High Perfrmance Data with VMware vfabric GemFire Systems using vfabric GemFire as an enterprise data management system, where speed and cnsistency f data is f critical t the peratins f the business, can benefit frm the tuning guidelines in this sectin. 5.1 JVM Memry Segments In Figure 8, the varius memry segments are shwn in the cnfiguratin f running ne vfabric GemFire n ne JVM and ne virtual machine. Figure 8. JVM Memry Segments f One JVM Running n One Virtual Machine VM Memry fr GemFire = Guest OS Memry + JVM Memry fr GemFire JVM Memry fr GemFire = JVM Max Heap (-Xmx value) + JVM Perm Size (-XX:MaxPermSize) + NumberOfCncurrentThreads * (-Xss) + ther mem 5.2 JVM Tuning and Best Practices Page 24 f 36

25 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 18 JVM Versin BP 19 Use Server Flag Descriptin Use JDK 1.6.0_24 r later. Always add the -server flag t the JVM cmmand that starts the GemFire server. On server class hardware mst JVMs default t this, but it is a best practice t explicitly turn it n. BP 20 Set Initial Heap Equal t Maximum Heap Set -Xms (initial heap) t -Xmx (maximum heap). Nte Withut this setting, perfrmance can suffer if the initial heap setting is nt adequate and the JVM respnds by increasing the memry allcated, causing verhead at runtime. BP 21 Garbage Cllectr Chice Set -XX:+UseCncMarkSweepGC t use the cncurrent lw-pause garbage cllectr and the parallel yung generatin cllectr. The lw-pause cllectr sacrifices sme thrughput t minimize stp-the-wrld GC pauses fr tenured cllectins. It requires mre headrm in the heap, s increase the heap size t cmpensate. BP 22 Disable Calls t System.gc() Set -XX:+DisableExplicitGC t disable full garbage cllectin. This causes calls t System.gc() t be ignred, aviding the assciated lng latencies. BP 23 Occupancy Fractin and Tenured Generatin Set -XX:CMSInitiatingOccupancyFractin=50 r even lwer fr high thrughput latency-sensitive applicatins that generate large amunts f garbage in the tenured generatin, such as thse that have high rates f updates, deletes, r evictins. This setting tells the cncurrent cllectr t start a cllectin when tenured ccupancy is at the given percentage. With the default setting, a high rate f tenured garbage creatin can utpace the cllectr and result in OutOfMemryErrr. T lw f a setting can affect thrughput by ding unnecessary cllectins, s test t determine the best setting. Page 25 f 36

26 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 24 New Generatin Size Descriptin Set the -Xmn value t be large enugh t avid the new generatin filling up. Making the new generatin large enugh avids the cst f cpying bjects int tenured space which can impact perfrmance. A cmmn apprach is t set the -Xmn size t apprximately 33% f the heap s maximum size, that is, 33% f -Xmx fr heap sizes less than 8GB. Fr heap sizes frm 8GB 100GB, the 33% rule might be t high typically 10%-15% f the maximum heap size fr these is adequate. T establish the best size fr yur applicatin yu must lad test and measure hw ften the new generatin fills up. Based n this, decide whether t adjust the -Xmn value. Prescribed range fr smaller heaps: if the heap is less than 8GB, -Xmn < 33% f -Xmx. Prescribed range fr larger heaps: if the heap is much greater than 8GB, then -Xmn < 10% t 15% f -Xmx. Hwever, it is difficult t apply this rule acrss the bard n varied wrklad behavirs. Fr example, with a very large heap, much greater than 8GB, it is ften fund that instead f using a percentage calculatin, yu can chse t cap it at 2GB. Then yu can prgressively make adjustments based n hw quickly yu see the yung generatin fill-up and hw it is impacting perfrmance. A further rule t cnsider when sizing -Xmn is that typically partitined regins mst likely have higher numbers f shrt lived bjects and therefre can require a larger -Xmn value. Cmpare this with replicated regins where the rate f change is minimal and nt many shrt lived bjects are created, and thus a lwer -Xmn is adequate. Nte This guidance has a caveat that depends n the behavir f the applicatin. Fr example, if the applicatin requires many Query calls, cnfigure a size at the upper end f the abve prescribed -Xmn range. Fr mstly Put calls, cnfigure a size in the middle f the range. BP 25 Using 32-Bit Addressing in a 64-Bit JVM When memry is cnstrained, set the -XX:CmpressedOps JVM ptin. This uses a 32-bit pinter address space within a 64-bit JVM fr heap sizes up t 32GB. This can save substantial memry space, in sme cases up t 50%, althugh the savings varies based n the Java types used within the applicatin. Nte Althugh this best practice saves n memry space usage, it can lwer the thrughput and increase latency. Page 26 f 36

27 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 26 Stack Size BP 27 Perm Size BP 28 Regin Placements in a JVM Descriptin In mst cases the default -Xss stack size is t large, ranging in size frm 256KB 1MB depending n the JVM and perating system. Reduce the stack size t cnserve the memry in use by the Java prcess. Fr example, yu can set -Xss192k. Althugh yu can experiment t determine what is best fr yur envirnment, yu might reduce this t a pint where a stack verflw ccurs, in which case yu have reduced it t far. At that pint yu shuld increase it t the pint where the stack verflw exceptin n lnger ccurs during the lad test. It is a cmmn best practice t set -XX:MaxPermSize in the range frm 128MB 256MB, althugh the actual size can vary fr yur applicatin, and apprpriate testing shuld be cnducted. The PermSize is where class level infrmatin is kept. In the HtSpt JVM this is allcated utside the heap, that is, in additin t the -Xmx. Place bth replicated and partitined regins within ne JVM instance; that is, within ne data management nde. This prvides the best scalability and perfrmance. By default, GemFire partitins each data pint int a bucket using a hashing plicy n the key. The physical lcatin f the key-value pair is abstracted away frm the applicatin. Additinally, there is n default relatin between data stred in different partitined regins. T run transactins n partitined regins, yu must clcate all data accessed in any single transactin n the same data hst. Additinally, in many cases, yu can get better perfrmance if yu clcate similar data within a regin and between regins. Fr example: A query run n a patient, her health recrds, and her insurance and billing infrmatin is mre efficient if all f the data is gruped in a single JVM. A financial risk analytical applicatin runs mre quickly if all trades, risk sensitivities, and reference data assciated with a single instrument are lcated tgether. Clcatin generally imprves the perfrmance f data-intensive peratins. Yu can reduce netwrk hps fr iterative peratins n related datasets. Yu can usually significantly increase verall thrughput fr cmpute-heavy, data-intensive applicatins. Yu specify clcatin thrugh cnfiguratin attributes and custm cding. Yu can: Add custm partitining cde t a regin t rute lgically cnnected data int the same buckets. Clcate multiple regins, s buckets with the same number in each regin are stred in the same JVM. Yu can verride the default vfabric GemFire mechanism t prvide yur wn clcatin. Refer t infrmatin abut PartitinReslver in the vfabric GemFire User s Guide. Page 27 f 36

28 High Perfrmance Data with VMware vfabric GemFire BP 29 vfabric GemFire Develpment and API Best Practices The vfabric GemFire prduct is rich in features. Applicatin develpers must be familiar with the fllwing tpics that are dcumented in the vfabric GemFire User s Guide. Quick Start Guide GemFire Member Data Regins and Operatins Regin Management Client/Server Cnfiguratin Handling Events Delta Prpagatin Querying and Indexing Cntinuus Querying GemFire Transactins Functin Executin Managing Member Relatinships Cache.xml Page 28 f 36

29 6. vfabric GemFire n VMware Best Practices 6.1 Overview High Perfrmance Data with VMware vfabric GemFire vfabric GemFire and vsphere prvide a rbust cmplement t deliver data faster and mre reliably using cst effective x86 cmmdity hardware and vsphere. If virtualizing vfabric GemFire, yu can leverage the best practices discussed in the Enterprise Java Applicatins n VMware. Best Practice BP 30 Enterprise Java Applicatins n VMware Best Practices Guide Descriptin The best practices discussed in this guide als apply t vfabric GemFire. Refer t the Enterprise Java Applicatins n VMware Latency Sensitive Applicatins Best Practices if Virtualized There are certain Guest OS, and netwrk cnsideratins that especially pertain t virtualized wrklads such as vfabric GemFire. These can benefit frm the fllwing best practices. Best Practice BP 31 Guest OS Descriptin Red Hat Enterprise Linux 5 and prir versins incur higher virtualizatin verhead due t high frequency timer interrupts, frequent access t virtual PCI devices fr interrupt handling, and an inefficient Linux timekeeping mechanism. By selecting a mre current versin f Linux, such as SUSE Linux Enterprise Server 11 SP1 r Red Hat Enterprise Linux 6 based n Linux kernels, r Windws Server 2008, these causes f virtualizatin verhead are minimized. In particular, Red Hat Enterprise Linux 6 has a tickless kernel that des nt rely n a high frequency interrupt-based timer, and is therefre much friendlier t virtualized latency-sensitive wrklads. Refer t Timekeeping best practices fr Linux guests ( and Timekeeping in VMware Virtual Machines ( Page 29 f 36

30 High Perfrmance Data with VMware vfabric GemFire BP 32 Physical NIC Mst 1GbE r 10GbE Netwrk Interface Cards (NICs) supprt a feature called interrupt mderatin r interrupt thrttling, which calesces interrupts frm the NIC t the hst s that the hst des nt spend all its CPU cycles prcessing interrupts. Hwever, fr latency-sensitive wrklads, the time that the NIC delays the delivery f an interrupt fr a received packet r fr a packet that has successfully been sent n the wire, is time adding t the latency f the wrklad. Mst NICs als prvide a mechanism, usually with the ethtl cmmand, t disable interrupt calescing. VMware recmmends t disable physical NIC interrupt mderatin n the VMware ESXi hst as fllws: # ethtl -C vmnicx rx-usecs 0 rx-frames 1 rx-usecs-irq 0 rx-framesirq 0 Where vmnicx is the physical NIC as reprted by the ESXi cmmand: # esxcli netwrk nic list Yu can verify that yur settings have taken effect by issuing the cmmand: # ethtl -c vmnicx Nte that althugh disabling interrupt mderatin n physical NICs is extremely helpful in reducing latency fr latency-sensitive virtual machines, it can lead t sme perfrmance penalties fr ther virtual machines n the ESXi hst, as well as higher CPU utilizatin t deal with the higher rate f interrupts frm the physical NIC. Disabling physical NIC interrupt mderatin can als defeat the benefits f Large Receive Offlads (LRO), because sme physical NICs (such as Intel 10GbE NICs) that supprt LRO in hardware autmatically disable it when interrupt mderatin is disabled, and the ESXi implementatin f sftware LRO has fewer packets t calesce int larger packets n every interrupt. LRO is an imprtant fflad fr driving high thrughput fr large message transfers at reduced CPU cst, s this trade ff shuld be cnsidered carefully. Additinal details are available in the knwledge base article, Pr TCP perfrmance can ccur in Linux virtual machines with LRO enabled: kb.vmware.cm/kb/ If the ESX hst is restarted the abve cnfiguratins must be reapplied. Page 30 f 36

31 High Perfrmance Data with VMware vfabric GemFire BP 33 Virtual NIC ESXi virtual machines can be cnfigured t have ne f the fllwing types f virtual NICs: Vlance, VMXNET, Flexible, E1000, VMXNET2 (Enhanced) r VMXNET3. This is described in Chsing a netwrk adapter fr yur virtual machine ( Use VMXNET3 virtual NICs fr yur latency-sensitive r therwise perfrmance critical virtual machines. It is the latest generatin f paravirtualized NICs designed fr perfrmance, and is nt related t VMXNET r VMXNET2 in any way. It ffers several advanced features including multiqueue supprt, Receive Side Scaling, IPv4/IPv6 fflads, and MSI/MSI-X interrupt delivery. Mdern enterprise Linux distributins based n r newer kernels, like Red Hat Enterprise Linux 6 and SUSE Linux Enterprise Server 11 SP1, ship with built-in supprt fr VMXNET3 NICs, s it is unnecessary t install VMware Tls t get VMXNET3 drivers fr these guest perating systems. VMXNET3 by default als supprts an adaptive interrupt calescing algrithm, fr the same reasns that physical NICs implement interrupt calescing. This virtual interrupt calescing helps drive high thrughputs t virtual machines with multiple vcpus with parallelized wrklads (fr example, multiple threads), while als striving t minimize latency f virtual interrupt delivery. Hwever, if the wrklad is extremely sensitive t latency, then VMware recmmends that yu disable virtual interrupt calescing fr VMXNET3 virtual NICs as fllws. Use VMware Prgrammatic APIs t add the special virtual machine cnfiguratin ptins as defined in VMware vsphere Web Services SDK Dcumentatin ( Refer t the VMware vsphere API Reference Dcumentatin, under the VirtualMachine Managed Object Type, fr the OptinValue[] extracnfig prperty f the VirtualMachineCnfigInf cnfiguratin prperty. T d s manually, first pwer ff the virtual machine. Edit yur virtual machine s.vmx cnfiguratin file and lcate the entries fr VMXNET3, as fllws: ethernetx.virtualdev = vmxnet3 ethernetx.calescingscheme = "disabled" Pwer n the virtual machines fr the virtual interrupt calescing settings t take effect. Nte This new cnfiguratin ptin is available nly in ESXi 5.0. Page 31 f 36

32 High Perfrmance Data with VMware vfabric GemFire 6.3 Memry Sizing f Virtual Machines Running vfabric GemFire As described in Enterprise Java Applicatins n VMware ( setting an apprpriate memry reservatin is essential fr Java based wrklads. Unique t GemFire wrklads, it is always advisable t have at least 50% headrm fr ptimal peratin. Refer t Figure 9. Figure 9. Virtual Machine Sizing Example: One GemFire Server Running n One JVM and One Virtual Machine VM Memry fr GemFire = Guest OS Memry + JVM Memry fr GemFire JVM Memry fr GemFire = JVM Max Heap (-Xmx value) + JVM Perm Size (-XX:MaxPermSize) + NumberOfCncurrentThreads * (-Xss) + ther mem Page 32 f 36

33 High Perfrmance Data with VMware vfabric GemFire Using the VM memry elements f the equatin in Figure 9 with the fllwing settings, the sizing assumptins are as fllws,: Where: Guest OS Memry is apprximately 0.5GB-1GB (depends n OS/ther prcesses). -Xmx is the JVM max heap size. -Xss is the Java thread stack size. The default is OS and JVM dependent, and can range frm 256kt 1MB. Perm Size is an area additinal t the Xmx (Max Heap) value and is nt GC-ed because it cntains class-level infrmatin. ther mem is additinal memry required fr NIO buffers, JIT cde cache, classladers, Scket Buffers (receive/send), JNI, GC internal inf. VM Memry fr GemFire = Guest OS memry + JVM Memry fr GemFire Let s assume that thrugh lad testing a JVM max heap (-Xmx) f 29696m has been determined as necessary. This max heap is made f actual memry usage f regin data that was determined t be 19797m, and applying the best practice f adding 50% head rm, this translates t actual data usage m head rm, which implies a ttal f 29696m. Therefre, -Xmx wuld be set t 29696m. Prceed t size as fllws: Set Xmx29696m and Xms29696m. Set XX:MaxPermSize=256m, which is a cmmn number and depends n the memry ftprint f the Class level infrmatin within yur Java applicatin cde base. The ther segment f NumberOfCncurrentThreads*(-Xss) depends largely n the NumberOfCncurrentThreads the JVM will prcess, and the Xss value yu have chsen. A cmmn range f Xss is 128k-192k. -Xss is OS and JVM dependent. If the stack is nt sized crrectly yu will get a StackOverflw. The default value is smetimes quite large and yu can benefit frm sizing it dwn t help save n memry cnsumptin. If fr example NumberOfCncurrentThreads is 100, then 100*192k => 19.2m (assuming yu set Xss t 192k) Assume the OS has a requirement f abut 500MB t run as per the OS spec. Ttal JVM memry (Java prcess memry) = 29696m (-Xmx) + 256m ( XX:MaxPermSize) + 100*192k (NumberOfCncurrentThreads*-Xss) + ther mem. Therefre JVM memry apprximately equals 29696m+256m+19.2m + ther mem = 29971m + ther mem Typically, ther mem is nt significant. Hwever, it can be quite large if the applicatin uses lts f NIO buffers and scket buffers. This value can be apprximated with ample space as abut 5% f the heap. That is, 5% * 29696= 1484m, althugh prper lad testing shuld be cnducted t verify. This implies that JVM prcess memry = 29971m+1484m=31455m. T determine the VM memry, assume yu are using Linux with n ther significant prcess running n it (nly this single Java prcess), the ttal cnfigured memry fr the virtual machine translates t: VM memry fr GemFire Server = 31955MB + 500MB = 31955m Page 33 f 36

34 High Perfrmance Data with VMware vfabric GemFire Set the VM memry as the memry reservatin. Yu can chse t set the memry reservatin as 31955m, but ver time yu shuld mnitr the active memry used by the virtual machine that huses this JVM prcess and adjust the memry reservatin t that active memry value, which culd be less than 31955m. This als means that the NUMA rules apply and yu want t make sure that each scket n the server has at least 32GB f RAM t huse this virtual machine, alng with the vcpus needed. Best Practice BP 34 Set Memry Reservatin and Allw fr 50% f Memry Headrm Descriptin If yur GemFire regin data requires a GemFire server t have 19797m f memry, allw fr 50% headrm fr ptimal perfrmance. This implies 19797m m (50% verhead), therefre a ttal f 29696m. Fllwing the sizing example shwn in Figure 9, set the memry reservatin t apprximately 31955m. Setting a memry reservatin directs that the reserved physical memry is made available by VMware ESX r ESXi t the virtual machine when it starts. D nt vercmmit memry. When sizing memry fr vfabric GemFire server within ne JVM n ne virtual machine, the ttal reserved memry fr the virtual machine shuld nt exceed what is available within ne NUMA nde fr ptimal perfrmance. Refer t BP 35 fr further NUMA discussin and cnsideratins. Page 34 f 36

35 High Perfrmance Data with VMware vfabric GemFire 6.4 vcpu sizing f Virtual Machines Running vfabric GemFire Best Practice BP 35 Enable Hyper- Threading and D Nt Overcmmit CPU Descriptin Always enable hyper-theading. D nt vercmmit CPU as vfabric GemFire are typically latency-sensitive applicatins that are CPU-bund. Size these based n the available physical cres. Fr mst prductin vfabric GemFire servers, size with a minimum f tw vcpu virtual machines. Hwever, larger 8-vCPU virtual machines might be necessary in sme cases t achieve yur SLAs. Encurage gd NUMA lcality by sizing virtual machines t fit within the NUMA nde. If yu suspect yur virtual machine has pr NUMA lcality, inspect the N%L cunter frm esxtp. With gd NUMA lcality, this cunter shuld be 100%. Nte A NUMA nde is equivalent t ne CPU scket, s fr a server with tw sckets there are tw NUMA ndes. Therefre, the available number f physical CPU cres and RAM is divided equally amng the NUMA ndes. This is critical when sizing virtual machines t fit within ne NUMA nde. Fr example, a 2-scket 16-cre (8 cres n each scket) server with 192GB RAM, has tw NUMA ndes, each with 8 physical cres (CPUs) and 96GB RAM (192/2). When sizing virtual machines it is imprtant nt t exceed the limits f 8 vcpus and 96GB RAM. Exceeding any f these CPU and RAM maximums f each NUMA nde can frce the virtual machine t fetch memry frm a remte lcatin, impacting perfrmance. There are many ESX/ESXi CPU scheduler enhancements t avid this, but fllwing this example can help. Yu can set sched.cpu.vsmpcnslidate = "true", as described in the knwledge base article, Cnslidating vcpus fr an SMP virtual machine can imprve perfrmance fr sme wrklads ( This instructs the ESX scheduler t place the vcpus f an SMP virtual machine int the fewest Last Level Cache (LLC) pssible. This plicy benefits by prviding better cache sharing. BP 36 Cache Server, JVM and VM Rati Have ne JVM instance per virtual machine. Typically, this is nt a requirement. Hwever, because vfabric GemFire JVMs can be quite large (up t 100GB), it is advisable t adhere t this rule in this case. Increasing the heap space t service mre data demand is better than installing a secnd instance f a JVM n a single virtual machine. If increasing the JVM heap size is nt an ptin, then cnsider placing the secnd JVM n a separate newly created virtual machine, thus prmting mre effective hrizntal scalability. As yu increase the number f GemFire servers, als increase the number f virtual machines t maintain a 1:1:1 rati amng the GemFire server, the JVM, and the virtual machines. Size fr a minimum f tw vcpu virtual machines with ne GemFire server running in ne JVM instance. This allws ample CPU cycles fr the garbage cllectr, and the rest fr user transactins. Page 35 f 36

36 High Perfrmance Data with VMware vfabric GemFire BP 37 VM Placement BP 38 vmtin, DRS Cluster, and GemFire Server Because vfabric GemFire can place redundant cpies f cached data n any virtual machine, it is pssible t inadvertently place tw redundant data cpies n the same ESX/ESXi hst. This is nt ptimal if a hst fails. T create a mre rbust cnfiguratin, use VM1-t-VM2 anti-affinity rules t indicate t vsphere that VM1 and VM2 can never be placed n the same hst because they hld redundant data cpies. When yu first cmmissin the data management system, place VMware Distributed Resurce Scheduler (DRS) in manual mde t prevent an autmatic VMware vsphere vmtin migratin that can impact respnse times. vmtin can cmplement vfabric GemFire features during scheduled maintenance t help minimize dwntime impact due t hardware and sftware upgrades. It is a best practice t trigger vmtin migratins ver a 10GbE netwrk interface t speed up the vmtin prcess. D nt allw vmtin peratins with vfabric GemFire lcatr prcesses as the latency intrduced t this prcess can cause members f the vfabric GemFire servers t falsely suspect that ther members are dead. Use DRS clusters dedicated t vfabric GemFire. If this is nt an ptin and GemFire has t run in a shared DRS cluster make sure that DRS rules are set up that will nt use vmtin t migrate vfabric GemFire virtual machines. Nte In sme cases a vmtin migratin might nt succeed and instead fails back due t a rapidly changing vlatile memry space, which can be the case with Partitined regins and in sme cases f Replicated regins. The failback is a fail-safe mechanism t the surce virtual machine and it des nt impact the surce virtual machine. vmtin makes this failback decisin based n the time it takes t cmplete the iterative cpy prcess that captures changes between the surce virtual machine t the destinatin virtual machine. If the changes are t rapid and vmtin is nt able t cmplete the iterative cpy within the default 100 secnds, it checks whether it can failsafe t the running surce virtual machine withut interruptin. Therefre, vmtin nly transfers the surce virtual machine t the destinatin if it is certain that it can cmplete the memry cpy. BP 39 - VMware HA and vfabric GemFire VMware HA shuld be disabled n vfabric GemFire virtual machines. If this is a dedicated vfabric GemFire DRS cluster, yu can disable HA acrss the cluster. Hwever, if this is a shared cluster, it is imprtant t exclude vfabric GemFire virtual machines frm HA. Nte Set up anti affinity rules between the vfabric GemFire virtual machines that will nt cause any tw GemFire servers t run n the same ESX hst within the DRS cluster. Page 36 f 36