VMware vcloud Architecture Toolkit High Performance Data with VMware vfabric GemFire Best Practices Guide

Transcription

1 VMware vclud Architecture Tlkit High Perfrmance Data with VMware vfabric GemFire Octber 2011

2 High Perfrmance Data with VMware vfabric GemFire This prduct is prtected by U.S. and internatinal cpyright and intellectual prperty laws. This prduct is cvered by ne r mre patents listed at VMware is a registered trademark r trademark f VMware, Inc. in the United States and/r ther jurisdictins. All ther marks and names mentined herein may be trademarks f their respective cmpanies. VMware, Inc 3401 Hillview Ave Pal Alt, CA Page 2 f 36

3 Cntents High Perfrmance Data with VMware vfabric GemFire 1. Intrductin Overview Purpse Target Audience Scpe References vfabric GemFire Architecture Overview vfabric GemFire Tplgies vfabric GemFire General Administratin Guide Overview Installatin Cnfiguratin Mnitring General Administratin and Trubleshting vfabric GemFire and Spring High Level Tuning Overview... Errr! Bkmark nt defined. 5.2 JVM Memry Segments JVM Tuning and Best Practices vfabric GemFire n VMware Best Practices Overview Latency Sensitive Applicatins Best Practices if Virtualized Memry Sizing f Virtual Machines Running vfabric GemFire vcpu sizing f Virtual Machines Running vfabric GemFire Page 3 f 36

4 High Perfrmance Data with VMware vfabric GemFire Page 4 f 36

5 1. Intrductin High Perfrmance Data with VMware vfabric GemFire 1.1 Overview This High Perfrmance Data with VMware vfabric GemFire prvides infrmatin abut best practices fr the deplyment f data fabric systems. The guide describes the best practices fr VMware vfabric GemFire data caching systems and its varius design cnstructs. The dcument captures fur main deplyment patterns cmmnly used t implement enterprise data requirements: vfabric GemFire deplyed as an enterprise data management system. vfabric GemFire deplyed as L2 cache. vfabric GemFire deplyed fr HTTP sessin management. vfabric GemFire deplyed as a faster mass data mver fr example, real-time reprting. 1.2 Purpse This guide prvides best practice guidelines fr deplying vfabric GemFire. The recmmendatins in this guide are nt specific t any particular set f hardware r t the size and scpe f any particular implementatin. The best practices in this dcument prvide guidance nly and d nt represent strict design requirements because enterprise data requirements can vary frm ne implementatin t anther. Hwever, the guidelines d frm a gd fundatin n which yu can build many f ur custmers have used these guidelines t successfully implement an enterprise data fiber fr their enterprise applicatins. 1.3 Target Audience This guide assumes a basic knwledge and understanding f vfabric GemFire, data management cncepts, and virtualizatin with VMware vsphere. Architectural staff can use this dcument t gain an understanding f hw the system wrks as a whle as they design and implement varius cmpnents. Engineers and administratrs can use this dcument as a catalg f technical capabilities. 1.4 Scpe This guide cvers the fllwing tpics: vfabric GemFire Architecture This sectin prvides a high level best practice architecture fr varius tplgies that are part f the high perfrmance data slutin space. vfabric GemFire Best Practices This sectin cvers varius best practices pertaining t setting up a data fabric in prductin, and GemFire n vsphere best practice cnsideratins. vfabric GemFire Mnitring and Trubleshting Primer There are times when yu have t trublesht a particular vfabric GemFire applicatin prblem. vfabric GemFire is equipped with several tls such as GemFire Tl fr Mnitring (GFMn), Visual Statistics Display (VSD), and vsphere esxtp utilities which are very infrmative when trubleshting. vfabric GemFire FAQ In this sectin, we answer sme frequently asked questins abut the varius data fabric deplyments. Page 5 f 36

6 1.5 References It is recmmended that yu becme familiar with the fllwing dcumentatin: High Perfrmance Data with VMware vfabric GemFire vfabric GemFire User s Guide: Enterprise Java Applicatins n VMware Practices fr Perfrmance Tuning f Latency-Sensitive Wrklads in vsphere VMs Page 6 f 36

7 2. vfabric GemFire Architecture 2.1 Overview High Perfrmance Data with VMware vfabric GemFire vfabric GemFire is an in-memry distributed data management platfrm that can be spread acrss many virtual machines, JVMs, and GemFire servers t manage applicatin bjects. Using dynamic replicatin and partitining it ffers the fllwing features in the platfrm: data durability, reliable event ntificatin, cntinuus querying, parallel executin, high thrughput, lw latency, high scalability, cntinuus availability, and WAN distributin. The fllwing figure shws GemFire as the middle data tier that rchestrates data delivery frm the backend datastres t the cnsuming applicatins. As demand frm cnsuming applicatins increases, the middle tier data layer expands t apprpriately meet demand. Fr further persistence, resiliency data can be written behind t a backup stre like a relatinal database fr archival purpses. GemFire als prvides full persistence durability using its wn native shared-nthing persistence mechanism. Figure 1. vfabric GemFire Architecture Best Practice BP 1 Cmmn Distributed Data Platfrm Descriptin When data delivery is required t be at the highest speed pssible, if millisecnds and micrsecnds matter, setting up vfabric GemFire as an enterprise data fabric system is the crrect apprach. By ding s and as shwn in Figure 1, yu intrduce a cmmn data delivery and cnsumptin layer in-memry fr all enterprise applicatins data needs. This allws yu t benefit frm the scalability, availability, and speed f executin features f vfabric GemFire. Page 7 f 36

8 2.2 vfabric GemFire Tplgies High Perfrmance Data with VMware vfabric GemFire There are three main setup tplgies fr vfabric GemFire: client/server, peer-t-peer, and multisite. Each f these tplgies can be used standalne r cmbined t frm an extended, full featured distributed data management system Client/Server Tplgy In a client/server tplgy there are tw tiers, a client tier and a server tier. In Figure 1, the client and server tiers are depicted. The client tier cmmunicates with the server tier t search fr r update data bjects frm the server tier. In the client tier, standalne client caches 1, 2, 3, and 4 cmmunicate directly with the server tier. Figure 2. vfabric GemFire Client/Server Tplgy Page 8 f 36

9 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 2 Client/Server Tplgy Descriptin Client/server tplgy is the mst cmmnly used fr enterprise class applicatins. The client sends individual peratins t the server t update cached data, t satisfy a lcal cache miss, r t run an ad hc query. The server streams cache update and cntinuus query events t the client based n client subscriptins. Fr advanced tuning and increased thrughput capacity, yu can distribute the lad f netwrk traffic fr yur client/server traffic thrugh a different adapter than the peer-t-peer traffic by setting a server bind address. These gemfire.prperty lines specify different nn-default addresses fr the member: bind-address= , server-bind-address= Use client/server tplgy when any f the fllwing are requirements f an enterprise class applicatin: Dynamic server discvery The GemFire server lcatr utility dynamically tracks server prcesses and directs clients t new servers, giving clients indirectin frm explicit server infrmatin. Clients need t knw nly hw t cnnect t the lcatr services. They d nt need t knw where servers are running r hw many servers are available at any time. Server grups Yu can assign yur servers t lgical grups that yur clients can refer t in their cnnectin cnfiguratins. Fr example, yu might use grups t manually partitin yur data, with ne grup f servers hsting ne set f data and anther hsting anther set. Or yu might use a grup t direct all database-centric traffic t the subset f servers that are directly cnnected t a backend database. Servers can belng t multiple grups. Yur clients need t specify nly the grup t use and are islated frm having t knw which servers belng t which grups. Server lad balancing The GemFire server lcatr tracks current lad infrmatin fr all servers, directing new client cnnectins t the servers with the least lad. GemFire prvides a default lad prbe fr yur servers, which yu can replace with yur wn custmized plug-in. Server cnnectin cnditining Client cnnectins can be cnfigured t transparently time ut and be replaced with new cnnectins which allws verall server use t be rebalanced after new servers are started. This helps speed cnditining in situatins such as adding servers r recvery frm server crashes and ther dwntime. Autmated data and query updates Yur clients can subscribe t events in the server. These events can include data updates and updates t results fr cntinuus queries that the client has registered with the server. The server uses subscriptin queues t send the updates asynchrnusly. Server failver and high availability When servers crash, the client cnnectins autmatically fail ver t the remaining servers. If the servers are sending autmated updates t the clients, the update requests als autmatically fail ver. Yu can cnfigure redundancy in yur server subscriptin queues s that the failver des nt interrupt the stream f events frm the server side. Page 9 f 36

10 High Perfrmance Data with VMware vfabric GemFire BP 3 Client/Server Cmmn Sizes Because vfabric GemFire is hrizntally scalable, scalability is limited nly by the hardware resurces available. Cnfigure vfabric GemFire data management ndes t hst apprximately 1TB f data. While there is n prduct limit as t the number f JVMs deplyed in a data management system, up t 32 JVMs have been implemented in varius prductin systems. Thusands f clients can cnnect back t the data management systems accessing data. Client scalability can be managed thrugh cnnectin pls. Run within a 64-bit JVM, and a heap size f up t 32GB. When using -XX:CmpressedOps, the 32GB heap space uses 32-bit pinter addressing which saves large amunts f memry as ppsed t using 64-bit pinter addressing. With this apprach yu can cntinue t run inside a 64-bit JVM and benefit frm larger heap sizes, but with cmpressed pinter addressing. Any heap size beynd 32GB uses 64-bit pinter addressing because the -XX:CmpressedOps ptimizatin is limited t 32GB. This is limitatin in the Java ptimizatin and is nt specific t vfabric GemFire as GemFire supprts larger than 32GB heap sizes. Nte Client lcal cache generally starts with zer client side data and is enabled nly when needed fr perfrmance ptimizatin Peer-t-Peer In this tplgy, tw r mre intercmmunicating vfabric GemFire servers frm a distributed system. The data is distributed accrding t the data regin s cnfiguratin redundancy rules. Figure 3. Peer-t-Peer vfabric GemFire Distributed System Page 10 f 36

11 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 4 Peer-t-Peer Multihmed Machines Descriptin If running n multihmed machines, yu can specify a nn-default netwrk adapter fr cmmunicatin. In nn-multicast peer-t-peer situatins, cmmunicatin uses the bindaddress prperty. This address must be the same fr all vfabric GemFire servers within the distributed system. BP 5 Peer-t-Peer Sckets Highly cncurrent/high-thrughput deplyments need cnserve-sckets set t false and then limit the NIO thread pl servicing clients if (and nly if) the number f peer-t-peer sckets/wrker threads increases t the pint where cntext switching verhead degrades perfrmance. Mre ndes in the peer-t-peer cluster imply mre cnnectin/wrker thread verhead, and thus a pssible reasn t lwer the per-server NIO pl size. Cnversely, mre pwerful hardware with mre available cres, and running n a mre pwerful underlying netwrk fabric implies the ability t increase the per-server NIO pl size. Fr peer-t-peer threads that d nt share sckets, yu can use the scket-lease-time t limit the time that a scket sits idle. When a scket that belngs t an individual thread remains unused fr this time perid, the system autmatically returns it t the pl. The next time the thread needs a scket, it retrieves ne frm the pl. Scket-buffer-size determines the buffer size. Buffers shuld be at least as large as the largest stred bjects and their keys, plus sme verhead fr message headers. The verhead varies depending n wh is sending and receiving, but 32KB shuld be sufficient. Larger scket buffers allw yur members t distribute data and events mre quickly, but they als take memry away frm ther requirements. Nte This prvides excellent perfrmance even fr small update sizes, while nt killing the ptential fr larger-sized chunking t ptimize bulk peratins putall()/getall()/queries and rebalancing/failver/failback. BP 6 Peer-t-Peer hsts File BP 7 Use Lcatrs in Managed Peer-t-Peer Envirnments Verify that every peer-t-peer hst has a hsts file entry fr itself and fr all ther hsts n the LAN. The hsts file entry frmat shuld fllw hst + dmain (fr example, "gemserver1" and "gemserver1.vmware.cm") are present fr each IP address entered in the hsts file. Lcatrs using TCP/IP Using this methd yu run GemFire lcatr prcesses that manage the authritative list f active peer-t-peer distributed system members. These lcatrs are peer lcatrs. A new member cnnects t ne f the lcatrs t retrieve the member list which it uses t jin the system. Lcatrs are highly recmmended fr prductin systems. Fr prductin envirnments, always use at least tw lcatrs n different hsts. Page 11 f 36

12 Nte High Perfrmance Data with VMware vfabric GemFire The client/server tplgy is the mst cmmnly used in enterprise applicatins. There are sme rare cases when perfrmance cnstraints are s strenuus that a single hp is all that can be affrded t meet SLAs. Peer-t-peer tplgy typically has ne netwrk hp between peers as ppsed t client/server tplgy where there are tw netwrk hps, if yu assume that there are at least tw redundant servers with which a client can cmmunicate. Hwever, when using peer-t-peer tplgy it is assumed that the rich features f a client/server tplgy, such as cntinuus querying, registratin f interest, and cnnectivity thrugh pled cnnectins are nt needed. These features are available nly with the client/server tplgy. Client/server tplgy is the mst cmmnly used as it is the mst feature rich Multisite Tplgy In the case f multisite tplgy, as shwn in Figure 4, there are tw sites each with a distributed system. Within each site, ne server member is nminated as the gateway t prvide data distributin between sites in case f a failure event, r fr ther enterprise data distributin requirements. The Site 1 and Site 2 tplgy can lcate bth sites within ne datacenter, r the sites can be distributed gegraphically at different datacenters if needed. Figure 4. vfabric GemFire Multisite Tplgy Page 12 f 36

13 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 8 Multisite Tplgy Descriptin Use multisite tplgy in distributed data systems that require a rbust failver mechanism at the applicatin data layer. Use the cnflatin feature when using a gateway hub s that nly the latest updates are passed ver t the remte site. With cnflatin, earlier entry updates in the queue are drpped in favr f updates sent later in the queue. This is prblematic fr applicatins that depend n seeing every update. Fr example, if any remte gateway has a CacheListener that needs t knw abut every state change, yu shuld disable cnflatin. T enable cnflatin, yu can set the batch-cnflatin attribute t true within the gateway-queue cache cnfiguratin element. In a multisite installatin using gateways, messages can back up in the gateway queues if the link between sites is nt tuned fr ptimum thrughput. If a receiving queue verflws because f inadequate buffer sizes, it can becme ut f sync with the sender and the receiver is unaware f the cnditin. The gateway s <gateway> scket-buffer-size attribute shuld match the gateway hub s <gateway-hub> scket-buffer-size attribute fr the hubs the gateway cnnects t. Fr example: <gateway-hub id="eu" prt="33333"> <gateway id="us" scket-buffer-size="42000"> </gateway> </gateway-hub> <gateway-endpint id="us-1" hst="ushst" prt="11111"/> <gateway-queue verflw-directry="verflw" maximum-queue-memry="50" batch-size="100" batch-time-interval="1000"/> <gateway-hub id="us" prt="11111" scket-buffer-size="42000"> <gateway id="eu"> </gateway> </gateway-hub> <gateway-endpint id="eu-1" hst="euhst" prt="33333"/> <gateway-queue verflw-directry="verflw" maximum-queue-memry="50" batch-size="100" batch-time-interval="1000"/> Avid verflwing t disk when pssible by adjusting the maximum-queue-memry attribute t accmmdate needed memry. Hwever, shuld yu wish t verflw t disk, yu can easily d s t prvide additinal data reliability. Fr prductin systems and higher availability, set enable-persistence t true fr the gateway-queue attribute. This causes the gateway queue t persist t the disk stre specified in disk-stre-name. Althugh fr ease f illustratin we shw tw sites, typically yu wuld implement n+1 sites t achieve fault tlerance. The multisite tplgy can als span a WAN with multiple sites in, fr example, New Yrk, Tky, and Lndn. Refer t Figure 5. Page 13 f 36

14 Nte High Perfrmance Data with VMware vfabric GemFire Gateway hubs and gateways cmmunicate thrugh TCP/IP sckets. The gateway hub listens at a specified address and prt fr gateway cmmunicatin frm remte sites. Gateways are cnfigured with endpint infrmatin matching the remte gateway hub specificatins. The gateway sends cnnectin requests t the gateway hubs t establish tw-way TCP cnnectins. Fr infrmatin n the multisite cnfiguratin, refer t the Cnfiguring Multisite Installatins sectin f the vfabric GemFire User s Guide ( In additin t the site-t-site cmmunicatin, each gateway hub is a member in its wn distributed system. Figure 5 shws three glbal sites in New Yrk, Lndn, and Tky. Each site has a primary gateway and a backup gateway. It is imprtant t inspect and tune the cnfiguratin parameters f the WAN gateways. Figure 5. vfabric GemFire in a Glbal Multisite Cnfiguratin Page 14 f 36

15 2.2.4 Using vfabric GemFire as Simple L2 Cache High Perfrmance Data with VMware vfabric GemFire In Figure 6 a client/server tplgy is used t cnfigure vfabric GemFire as a Hibernate L2 cache. This cnfiguratin has the added benefits f faster perfrmance with relative ease f cnfiguratin. This is installed as a Hibernate plug-in and therefre n cde change is required. It als keeps the query results as distributed cache bjects thus imprving perfrmance and availability. Figure 6. Using vfabric GemFire as Hibernate L2 Cache Page 15 f 36

16 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 9 Hibernate L2 Cache Descriptin Turn n L2 cache in the Hibernate cnfiguratin (hibernate.cfg.xml): <prperty name="hibernate.cache.use_secnd_level_cache">true</prperty> Set regin.factry_class t GemFireReginFactry (hibernate.cfg.xml versin 3.3+): <prperty name="hibernate.cache.regin.factry_class"> cm.gemstne.gemfire.mdules.hibernate.gemfirereginfactry </prperty> Set the cache usage mde t: Read nly Used when yu d nt plan t mdify the data already stred in persistent strage. Read write Used when yu plan t bth read frm and write t data. Nn-strict write A special read/write mde that has faster write perfrmance. Use this nly if n mre than ne client updates the data at a time. Transactinal Allws fr transactin based data access. The cache mde can be set either using anntatin r in the Hibernate mapping file: T set using the Hibernate mapping file entity_name.hbm.xml: <hibernate-mapping package="package"> <class name="entity_name"...> </class> <cache usage="read-write nnstrict-read-write readnly"/>... </hibernate-mapping> T set the mde anntatins: imprt rg.hibernate.anntatins.cache; @Cache(regin = 'REGION_NAME', usage = CacheCncurrencyStrategy.READ_ONLY READ_WRITE NONSTRICT_READ_WRIT E TRANSACTIONAL) public class MyClass implements Serializable { }... Page 16 f 36

17 2.2.5 Using vfabric GemFire as an HTTP Sessin Cache High Perfrmance Data with VMware vfabric GemFire Best Practice BP 10 Tplgies fr HTTP Sessin Management Descriptin Either client/server, peer-t-peer, r multisite vfabric GemFire tplgies can be used t achieve HTTP sessin replicatin. If dealing with user sessin data that must be cmpletely fault tlerant, use multisite vfabric GemFire tplgy and HTTP sessin management. Fllw the recmmended setup in the HTTP Sessin Management Mdule sectin f the vfabric GemFire User s Guide ( It is relatively straightfrward with minimal change t cnfigure HTTP sessin replicatin with GemFire n VMware vfabric tc Server. In Figure 7, vfabric GemFire is used fr HTTP sessin replicatin that can be easily achieved when plugged int vfabric tc Server. Figure 7. Using vfabric GemFire fr HTTP Sessin Replicatin Using vfabric GemFire as a Faster Data Mver Best Practice BP 11 Real-Time Reprts Descriptin vfabric GemFire client/server tplgy is mst suited fr real time reprt setup. This allws yu t mve rapidly changing data t the cnsuming end pint client cache t present the data in real time. vfabric GemFire features such as cntinuus querying and functin executin can help in the implementatin f a business critical real time reprts. Page 17 f 36

18 High Perfrmance Data with VMware vfabric GemFire 3. vfabric GemFire General Administratin Guide 3.1 Overview The fllwing sectins summarize sme high level best practices. There are additinal details in the Administratin sectin f the vfabric GemFire User s Guide. 3.2 Installatin T dwnlad vfabric GemFire, g t Fllw the installatin instructins at: Cnfiguratin The mst ntable cnfiguratin files within vfabric GemFire are gemfire.prperties, gemfirelicenses.zip, and cache.xml. gemfire.prperties Cntains the settings required t jin a distributed system. Cnfiguratin includes system member discvery, cmmunicatin parameters, security, lgging, and statistics. Fr a detailed descriptin f the parameters within this file, refer t the vfabric GemFire User s Guide. gemfirelicense.zip The license file which shuld never be unzipped. Nte This is the license file fr vfabric GemFire 6.5. With vfabric GemFire 6.6 and later, licensing is dne using serial numbers. Refer t the vfabric GemFire User s Guide fr details. cache.xml The declarative cache cnfiguratin file. This file cntains XML declaratins fr cache, regin, and regin entry cnfiguratin. It is als used t cnfigure disk stres, database lgin credentials, server and gateway lcatin infrmatin, scket cnfiguratin, and s frth. Page 18 f 36

19 High Perfrmance Data with VMware vfabric GemFire Best Practice BP 12 Cnfiguratin Descriptin D nt t unzip the gemfirelicense.zip file leave it intact. Each f the three cnfiguratin files has a default name, a set f file search lcatins, and a system prperty that can be used t verride the defaults. T use the default file specificatin, place the file at the tp level f its directry r jar file. The system prperties are standard file specificatins that can have abslute r relative pathnames and filenames. If yu d nt specify an abslute file path and name, the search lks thrugh all the search lcatins fr the file. The gemfire.prperties file can be specified with the system-level Java prperty -DgemfirePrpertyFile=<valid file/path>. Yu can verride any GemFire prperty set in the file r by the CacheFactry API with a system-level Java argument that fllws the pattern -Dgemfire.<prperty-name>=<prperty-value>. Deply the same gemfirelicense.zip n all members f the peer-t-peer tplgy, fr vfabric GemFire 6.5. Fr vfabric GemFire 6.6 use the same license key n all members f the peer-tpeer tplgy. All peer-t-peer members f the distributed system must have the same versin f vfabric GemFire. Clients can be up t ne majr release behind. Fr example, any 6.x client interperates with any 6.x r 7.x server, but nt with an 8.x server. The vfabric GemFire prperty aut-start=true must be cnfigured fr the agents fr any versin f GemFire 6.5. Fr highly cncurrent wrklads, set the GemFire prperty cnservesckets=false n the data management ndes (DMNs). If the scale is large and t many sckets (and assciated threads t service thse sckets) are created between the DMNs, tune the CacheServer (a cnfiguratin element in the DMN cache.xml cnfiguratin) t reduce the NIO thread pl servicing client requests. This places a hard upper limit n the pssible number f DMN peer-t-peer cmmunicatin sckets. Refer t the vfabric GemFire User s Guide fr infrmatin: <cache-server max-cnnectins="<integer number>" </cache-server> max-threads="<integer number>"> Place the default files either in the current directry frm which yu start the GemFire server r n the CLASSPATH. If yu wish t change the default names f these cnfiguratin files, yu can set the fllwing prperties t verride them. Nte These prperties are useful t script the deplyment r mvement f the cde base frm Dev t QA, and then t prductin, where there might be a separate set f cnfiguratin files fr each envirnment. Depending n which envirnment is deplyed t yu can rtate the apprpriate files in the gemfireprpertyfile, gemfire.cache-xml-file, r gemfire.license-file. Set -Djava.net.preferIPv4Stack=true in the start script fr all servers, peers, and lcatrs. Page 19 f 36

20 3.4 Mnitring High Perfrmance Data with VMware vfabric GemFire Best Practice BP 13 vfabric GemFire Mnitring Tls Descriptin vfabric GemFire is a specialized prduct and it is imprtant that administratrs are familiar with the available mnitring tls. The vfabric GemFire Tls Guide ( details GFMn and Visual Statistics Display (VSD) tls available fr mnitring. The GFMn tl mnitrs a vfabric GemFire system in real time, prviding health infrmatin, detailed peratinal and cnfiguratin data, system alerts, thrughput perfrmance, and statistics fr system members and cnnected clients. The VSD tl reads GemFire statistics and prduces a graphical display fr analysis. Cnfigure the vfabric GemFire prperty statistics-enabled=true t generate statistics files that can be viewed with VSD. This can be critical t trublesht ptential prblem areas r help t diagnse a prblem. Yu can als use the VMware vfabric Hyperic GemFire plug-in that prvides a live data user interface fr viewing metrics in real time. Refer t the vfabric Hyperic Guide: Yu can als use esxtp t mnitr vsphere, refer t the trubleshting sectin f the Enterprise Java Applicatins n VMware General Administratin and Trubleshting Best Practice BP 14 General Administratin Descriptin Fr managing disk stres, security, system lgs, trubleshting, the Cmmand Line Utility, and fr administering the Distributed System, refer t the vfabric GemFire User s Guide. D nt cnfigure firewall restrictins between the LAN hsts. There must be least fur dedicated prts pen thrugh any intervening firewall (cnnecting t the peert-peer hsts): ne fr the lcatr prt, ne fr the server prt, ne fr the agent HTTP prt, and ne fr the agent RMI prt. If trubleshting a cnnectivity prblem, set lg-level=fine n all sides f the cnnectin. This shuld always include the lcatr, as it is the first pint f cntact fr cnnectivity. At a fine lg level, yu can immediately see whether a scket cnnectin is made, and if it is, why and where the cnnectin is rejected. After trubleshting it is imprtant t revert t the default lg level, r typically in prductin, t the lg-level=errr. When yu first begin t diagnse a ptential cnnectivity r general prblem with the system, start with telnet t test whether a remte lcatr, server, r agent prt is reachable. Page 20 f 36

21 High Perfrmance Data with VMware vfabric GemFire BP 15 General Trubleshting The vfabric GemFire User s Guide has a detailed sectin n Trubleshting and System Recvery. Fllw thse instructins. BP 16 Trubleshting SYN Ckies When trubleshting perfrmance prblems, check t see yu are nt impacted by SYN ckies. SYN ckies are the key element f a technique used t guard against SYN fld attacks. Daniel J. Bernstein, the technique's primary inventr, defines SYN ckies as particular chices f initial TCP sequence numbers by TCP servers. In particular, the use f SYN ckies allws a server t avid drpping cnnectins when the SYN queue fills up. Instead, the server behaves as if the SYN queue had been enlarged. The server sends back the apprpriate SYN+ACK respnse t the client but discards the SYN queue entry. If the server then receives a subsequent ACK respnse frm the client, the server is able t recnstruct the SYN queue entry using infrmatin encded in the TCP sequence number. T check fr the presence f SYN ckies: grep SYN /var/lg/messages Aug 2 12:19:06 w1-vfabric-g1 kernel: pssible SYN flding n prt Sending ckies. Aug 2 12:54:38 w1-vfabric-g1 kernel: pssible SYN flding n prt Sending ckies. Aug 3 10:46:38 w1-vfabric-g1 kernel: pssible SYN flding n prt Sending ckies. T determine whether r nt SYN ckies are enabled (1 is n, 0 is ff): $ cat /prc/sys/net/ipv4/tcp_synckies 1 T temprarily disable SYN ckies (changes at rebt): # ech 0 > /prc/sys/net/ipv4/tcp_synckies Permanently disable SYN ckies: Add\mdify the fllwing in /etc/sysctl.cnf # Cntrls the use f TCP synckies net.ipv4.tcp_synckies = 0 Page 21 f 36

22 4. vfabric GemFire and Spring High Perfrmance Data with VMware vfabric GemFire Best Practice BP 17 vfabric GemFire and Spring Descriptin Use Spring t cnfigure GemFire servers and regins rather than manually creating them with applicatin cde. This allws yu t centralize yur applicatin service cnfiguratin, as ppsed t having the Spring cntext cnfiguratin plus a separate cache.xml file. Yu can eliminate the need t implement Declarable n yur cache laders, cache listeners and cache writers. Spring takes care f binding these cmpnents int GemFire regins thrugh dependency injectin. These cmpnents can als be shared amng regins as singletns using Spring's nrmal DI techniques. Yu can leverage advanced techniques t cnfigure GemFire via SpEL, selectively expsed cnfiguratin parameters, and s n. Use the Spring GemFire prject t mve yur GemFire cnfiguratin int the Spring cntext mre easily. Refer t Spring GemFire Hme at Use the GemFire schema extensin t the Spring cntext cnfiguratin t simplify the cnfiguratin f the varius GemFire cmpnents further with validated cnfiguratin prperty names (Sectin 1.1 in the Spring GemFire dcumentatin). Use GemfireTemplate t simplify interactins with the GemFire APIs. GemfireTemplate includes many best practice techniques fr dealing with resurce management and multiple threads in a virtual machine wrking with GemFire. GemfireTemplate eliminates the need fr yu t deal with checked exceptins. The GemfireTemplate prvides the best practice technique fr ensuring thread safe access t GemFire resurces in a single virtual machine. GemfireTemplate prvides utility methds t access and manage data mre simply. Transactin management using Spring and Spring GemFire: This prvides a prtable, well integrated way t prvide transactins t yur applicatin t prmte well defined behavir fr multiple threads mdifying the data. Autmatically cnfigures the GemFire server with best practice recmmendatins fr safest use in a multithreaded envirnment by enabling cpyonread t prevent client threads frm inadvertently editing data cntents in a nn-transactinal way. Use InstantiatrFactryBean t autmatically generate an efficient Instantiatr: Reflectin is the default technique used t serialize and deserialize data acrss the entire distributed data management system. If data serializatin is a bttleneck in yur applicatin, the general best practice recmmendatin is t implement custm instantiatr lgic t speed up the serializatin prcess. The InstantiatrFactryBean takes a list f dmain types (and a unique integer ID fr each type t efficiently serialize type inf as an integer instead f a string) and autmatically generates instantiatrs using the ASM bytecde library that prevents having t use reflectin t serialize and deserialize bject data. Page 22 f 36