ID232 IBM Connections Deployment and Performance Planning

Transcription

1 ID232 IBM Connections Deployment and Performance Planning Hai Jie Wu Senior Software Engineer IBM 2012 IBM Corporation

2 IBM s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion IBM Corporation

3 Agenda Planning & Capacity Sizing Considerations User population Use cases Sizing Considerations Picking the Right Deployment Tuning Your Deployment Measuring Your Health Q&A IBM Corporation

4 How many users does IBM Connections need to support? Users will drive what sort of deployment is needed to run Connections Registered Users vs. Active Users vs. Concurrent Users Registered users: How many user records in your enterprise directory server? Active users: How many users will use Connections in a working day? Concurrent users: How many users are currently using Connections? No authoritative standard for defining active vs concurrent. How many users will actually use Connections every day after deployment? Based on statistics data of existing services, and estimate Reference other deployment Let's guess How are your users distributed? A worldwide distribution? Will they access Connections 7X24? Users in same time-zone but from different locations? Local intranet users? IBM Corporation

5 What will these users do with IBM Connections? Analyze and estimate based on average daily usage What Connections components will your users use from web browsers? What's the frequency of use per day per user? Is Connections being used to perform existing business processes? What metrics exist for those processes today? What else will your users be using? IBM Lotus Note integration? IBM Lotus WebSphere Portal integration? IBM Lotus Sametime plugins? Other connectors? Crawlers? Feeds in other applications? Some things to consider Will Homepage be the entry point? Will Profiles be your corporate white pages? Is Connections replacing one or more existing services? Different frequencies of web users and plugin users IBM Corporation

6 Sizing considerations Handle future growth User population growth Usage growth Disk volume growth Storage consideration Tip Go to IBM Techline team for sizing assistance Heavy usage on Files? Wikis? Community Media Gallery? When is your Peak Hour? Your system should sustain acceptable server response time during peak hour. Peak Load Peak load vs. average load in hour Average Load IBM Corporation

7 Agenda Planning & Capacity Sizing Considerations Picking the Right Deployment Connections Install Types Key Performance Factors Performance Considerations Picking the Right Deployment Tuning Your Deployment Measuring Your Health Q&A IBM Corporation

8 Which Install Type to Choose? Options available in the IBM Connections installer From 3.0, all install configurations are Network Deployment Small Footprint Deployment 1 Cluster hosting all features Medium Footprint Deployment Features grouped into several clusters, default grouping pre-chosen. Large Footprint Deployment Each feature installed into its own cluster IBM Corporation

9 General Recommendations of Install Type < 1000 registered users Small deployment Single machine is adequate 1K 10K registered users Medium deployment Separate out your database tier from your application server 10K 100K registered users Large deployment May need additional cores and memory, reverse proxy cache to off-load traffic. > 100K registered users Large deployment Use multiple nodes for scale out, multiple database servers, regional reverse proxy caches Tip Use Medium or Large Deployment for any non proof of concept. 64 bit OS is recommended IBM Corporation

10 Key Performance Factors Connections key performance factors CPU Utilization on Application Server Disk I/O Both on the database and file repository Network bandwidth Internal: Between the database and file server and the application server. External: Server to end user. Impact end user experience. Incorrect deployment topology: Inadequate server memory. e.g. [ put multiple components in one 32bit server node], or [put all components on one node with 4 GB of RAM] Inadequate database instance memory. e.g. [put multiple databases in one 32 bit database instance ] Inadequate total system resources. High Latency between user and server. [ w/out regional proxy] Unchecked System Health (improper tuning) No tuning or exact same tuning setting from tuning guide documentation IBM Corporation

11 Performance Considerations on Deployment Questions to ask ourself Do we have enough Disk I/O bandwidth? Is the system using a dedicated disk that's separate from the database server? Do we have adequate memory for database caching? Is the share file system for component storage & local file system for index performant? Do we have enough network bandwidth? Do we have a gigabit network between internal servers? Any remote connected servers from deployment? Is the network latency acceptable? e.g. LDAP. Have we optimized and tuned our network? What servers do we need? Server type? 64 OS vs. 32 bit OS? Do we need horizontal scaling or vertical scaling? Do we need a WebSphere Application Server Edge component as caching proxy server at front-end? Do we need to separate Connections components into different server instances? How to divide? IBM Corporation

12 Performance Considerations on Deployment What are the target user types? Internal users? External users? Mixed? What's their percentages? Global Deployment Considerations What is the latency between end user and server infrastructure? How does DNS work in my enterprise? Where would a remote proxy cache reduce latency for most impacted users? Does the custom theme have too many images? Could we use a sprite? Are they cached at the caching proxy server? Do we need solution with distributed regional proxy servers? How many database instances do we need? Do we need multiple database servers? Do we need High Availability(HA)? Disaster Recovery? IBM Corporation

13 Picking the Right Deployment (Medium: 64 bit + HA ) Deployment manager IBM directory Node 2 Database WebServer Activities Forums Profiles Wikis Activities Forums File Server Profiles wikis cluster 1 Medium deployment Two nodes for HA Deployment Manger node can be in any of the two nodes WebSphere Cell Node 1 Blogs Communities Bookmarks Files Blogs Communities Bookmarks Files cluster 2 Edge caching proxy server to off-load application server Mobile News Node Search 1 Mobile News Search cluster Infra cell Node 1 Node IBM Corporation

14 Picking the Right Deployment (Large:64 bit + HA ) Deployment manager IBM directory Node 2 Database servera servera cluster A Proxy Server WebServer serverb serverc serverd serverb File Server serverc serverd cluster B cluster C cluster D Large deployment Node 1 serverf serverf cluster F Two nodes for HA Requires more resources for each node. WebSphere Cell serverh serverm serverw serverh serverm serverw cluster H cluster M cluster W Node serverp 1 serverp cluster P Node 1 Node 2 cell IBM Corporation

15 Picking the Right Deployment (Extra-Large) Active IHS/WAS nodes Active DB2 IBM directory Enterprise Load Balancer Multi-purpose server WAS DMGR Enterprise Regional Caching Proxy WebSphere Load Balancer NFS server Passive IHS/WAS nodes NFS SAN Passive DB2 DB2 SAN SMTP server Connections IBM Corporation

16 Picking the Right Deployment HTTP Server to off-load the application server. All deployments should have at-least one. HTTP Server and Application Server can be deployed in same server node if it has adequate memory and CPU cores Edge Proxy could off-load application server Large and/or taxed deployments should have at-least one. Less expensive to maintain than additional WebSphere nodes. Serves cached static content and dynamic query load with public content Internal benchmarks show ~30% cache hit rates Distributed Edge Proxy could reduce network latency and network bandwidth requirements. Global deployments with high latency connections to the server should have at-least one IBM Corporation

17 Picking the Right Deployment Application Server One server instance can have all the Connections features or only some. Database Server Splitting database instances across multiple database servers for extra large deployment. Vendor choice: Oracle Database, Microsoft SQL Server? or DB2? A system skills question vs. a scale out question. Storage devices Always use at least a multi-disk RAID array and consider a SAN for higher I/O bandwidth Follow database server tuning guidelines for optimal disk topology For high availability, each of these tiers also require redundancy. IP-spraying and load balancing will be necessary. VM solution Count on additional resource overhead if using VM solution. Database disk sub system is busier if using VM solution IBM Corporation

18 Agenda Planning & Capacity Sizing Considerations Picking the Right Deployment Tuning Your Deployment System/Infrastructure/Network Tuning Application Server Tuning Page Load Time Tuning Measuring Your Health Q&A IBM Corporation

19 Connections Performance Tuning Tuning guide is released in public wiki IBM Lotus Connections 3.0 Tuning Guide [link] Common mistakes Misunderstand the cause of bad performance complained by users Page Load Time includes: Network Time + Server Response Time + Browser Render Time Understand the problem before tuning Improper tuning Tune the system without understanding the implications of what you re changing, e.g. just copy settings from tuning guide Don t look at the results of tuning Only tune one part of the system Improper performance testing Lack of preparation: no/few data population for testing It reports too good result Lack of workload design: only login/navigate/logout Not reflect real user load Script issues: Not parameterized, e.g. all scripts use same user Lead you to wrong IBM Corporation

20 Connections Performance Tuning How to do performance tuning? Bottlenecks can occur at any level Identify: Review the symptoms of performance issue, and determine potential root cause Understand your environment Know your dependencies Analyze: Weigh the options available to correct the performance issue and the potential impacts or costs of those options Modify: Make changes to the system's configuration to attempt to resolve the issue Measure: Review the effectiveness of the change Know your data flow Know your networking Know where your load lies Bottom-up tuning Infrastructure/network Server/Operating Systems WebSphere Application Server Database server HTTP/Reverse Proxy server Tip Make a baseline benchmark for initial system Make only one tuning change at a time Make a new baseline after change is valid IBM Corporation

21 System/Infrastructure/Network Tuning OS Performance Tuning - General Deactivate all services you do not need Remove all protocols and bindings you don't need Maintain and upgrade your systems to the latest OS fix pack level. Apply general OS performance tuning [link ] suggested by WebSphere Application Server documentation. No Screen-Savers - Just lock the console Screen-Savers can keep the CPU busy Properly configure Kernel shared memory (Linux /IBM System z ). See specific documentation for your chosen database platform. Enable hardware multi-threading where available to improves system capacity. Hyper-Threading (HT) on Intel processors Simultaneous Multi-threading (SMT) on Power-series processors. Consider Network traffic General Establish high bandwidth / low latency connections between critical points in your environment Networking varies widely among locations/users Share application requirements with network team in order to understand implications for bandwidth and latency IBM Corporation

22 Connections Performance Tuning Thread Pool Several places we can expand/limit thread pool settings Edge Server IBM HTTP Server WebSphere Application Server(WAS) plugin for IBM HTTP Server WebSphere Application Server: Web Container Thread Pool Bad story Recommended solution Define the max HTTP connections for your system. Set threshold number at front-end HTTP/Edge layer. Normally, we don't need to restrict the max number at WAS plugin for HTTP Server. Set proper Web Container thread pool number for each Connections services, limit the max load for each services. Unlimited/large max available connections Degrade whole system due to excessive memory or exhausted CPU. Distribute the load in a clustering environment. Too small max available connection Slow page response time, queuing and waiting IBM Corporation

23 Connections Performance Tuning - Memory Heap setting for WebSphere Application Server Enable Generational garbage collector. (-Xgcpolicy:gencon.) Gencon: Optimized for short lived objects in transactional work loads. Splits GC into two segments (nursery, tenured), auto sized by default. Tenured used for long lived objects. GC less frequently with gencon policy Nursery used for short lived objects. Smaller space to mark/sweep so shorter GC pauses. Enable verbose GC logs to monitor JVM heap usage, tune the heap size based on result. When setting the maximum JVM heap size pay particular attention that sufficient physical memory exists to support the value chosen - avoid swapping. Edge Proxy Server memory Memory limitation for Edge proxy server (32bit) Two directives mainly affect in Edge configuration file ibmproxy.conf CacheMemory: Specify the cache RAM MaxActiveThreads: Specify the maximum number of active threads Set both with large number will cause server out of memory issue under high load IBM Corporation

24 Connections Performance Tuning - Database JDBC Connection Pool Default value is 10. Too small for production deployment. No magic #, based on usage patterns via monitoring. Too low will cause failed transactions. Too high will consume excess memory. Suggested starting min/max values to use for heavy load Prepared Statement Cache The default prepared statement cache size of 100 is optimum for DB2. Deployments with a different RDBMS, such as Oracle, however, which use larger prepared statements, you should consider reducing the size of the cache For large deployment, customize database creation to separate transaction logging from data and indexes IBM Corporation

25 Connections Performance Tuning Page Load Time Enable content compression and proper cache control header for static contents can greatly improve the page load time, especially via long distance low bandwidth network. Enable Content Compression Enable content compression at HTTP server Enable content compression at Edge proxy server Append Cache Control headers Static contents already set cache headers out-ofbox from Connections 3.0 Append Cache Headers is a good practice if with customized code Do a client side measurement before goes live! Easy to test, can do it manually Do test from real network conditions, long distance, different geographies Use tool to capture the page load time and requests. Analysis and fix client side performance issues IBM Corporation

26 Connections Performance Tuning - Miscellaneous HTTP Server Solutions vary by platform, read HTTP Server Performance Tuning[link] before changing. Configuring Files and Wikis downloading for production deployments[link] Edge proxy server The caching proxy configuration file, ibmproxy.conf, contains a number of important directives that affect performance, read product document in conjunction with tuning guide. Enable query request pattern caching, set to cache these responses if they contained a public header. CacheQueries Public Inter-service configuration Configure to avoid unnecessary round-trip requests for inter-services calls. VM specific tuning Count resource overhead for VM, assign adequate resource to VMs. Take more consideration into disk subsystem configuration including hardware topology and partition layout Enable Extended Page Support (EPT) if CPU supports the feature A physical NIC adapters dedicated for all VMs IBM Corporation

27 Agenda Planning & Capacity Sizing Considerations Picking the Right Deployment Tuning Your Deployment Measuring Your Health OS Monitoring Database Monitoring Application Server Monitoring HTTP Server Monitoring Q&A IBM Corporation

28 Tip Measuring your health Tracking and trending Choose what monitoring approach suits your organization best: Re-active approach: Identifies problems as they happen Pro-active approach: Requires more up-front effort, identifies potential issues before users are affected Track Key Performance Indicators in your infrastructure Understand what trends indicate bottlenecks under development Understand how trends relate to the normal fluctuations in user activity When does normal peaks develop What does usage patterns look like between peaks Pro-actively watch for Trends indicating an increase in load Trends indicating a plateaus have been reached immediately after an increase Choosing Technology Vendor Provided Tools or Solutions Often fragmented and require automation for end-to-end solution Open Source Tools Often fragmented and require automation for end-to-end solution Some require significant subject matter expertise IBM Tivoli Products IBM Tivoli Monitoring IBM Tivoli Composite Application Management (ITCAM) IBM Corporation

29 OS Monitoring Key Metrics Tools CPU (Per CPU and Total) Windows : Performance Monitor Average Utilization (%) Peak Utilization (%) Linux /AIX : NMON tool (see resources page) Memory Memory Free (Bytes) Memory Consumed (Bytes) Paging Disk IO (Per Disk/LUN and Total) Disk Busy (%) Disk Reads/Writes (Bytes/Sec) Average Disk Queue Length Disk Transfers Network (Per Interface and Total) Read/Write (Bytes/Sec) IBM Corporation

30 Database monitoring DB2 Metrics Database Snapshots Connection Pools DB2 Performance Expert Queries Oracle CPU consumption / execution time Oracle Enterprise Manager Console Microsoft SQL Server MS SQL Server Counters for Performance Monitor Execution count Queries & Bufferpools Physical vs Logical Reads Rows read Performance Dashboard Rows written Memory used per database / process IBM Corporation

31 Application Server Monitoring Threads Enable PMI Periodically inspect trends with Tivoli Performance Viewer WebContainer Threads Data Source Connections Heap Have verbose GC enabled Symptoms to start digging: Excessive CPU utilization? Long pauses in overall system responsiveness? Questions to answer: How much time is spent in GC? (is it > 15%? ) How much free space left (used heap)? (is it < 20 %? ) How much memory is freed after collection? Tool IBM Support Assistant Healthy: Unhealthy: IBM Corporation

32 HTTP Server Monitoring What response times are users experiencing Include %T (seconds) or %D (milliseconds) in LogFormat directive in httpd.conf Eg: LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T/%D" combined See access_log for metric output: [31/Jan/2008:14:19: ] "GET / HTTP/1.1" "" "Mozilla/5.0 (X11; U; Linux i686; en-us; rv: ) Gecko/ Firefox/ " 0/95491 How many HTTP worker threads are used? Enable status module in httpd.conf mod_status See for metric output "_" Waiting for Connection, "S" Starting up, "R" Reading Request, "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup, "C" Closing connection, "L" Logging, "G" Gracefully finishing, "I" Idle cleanup of worker, "." Open slot with no current process mod_mpmstats See access_log for metric output [Thu Aug 19 14:01: ] [notice] mpmstats: rdy 712 bsy 312 rd 121 wr 173 ka 0 log 0 dns 0 cls IBM Corporation

33 Resources Techline Capacity Sizing Contact your IBM Representative Connections Wiki Connections Benchmark Report Contact your IBM Representative Connections Tuning Guide WebSphere Application Server Performance Tuning Configuring Files and Wikis downloading for production deployments nmon performance: A free tool to analyze AIX and Linux performance HTTP Server Tuning Edge Caching Proxy Administration Guide IBM Support Assistant Garbage Collection and Memory Visualizer (GCMV) Tutorial topic=/com.ibm.iea.was_v7/was/7.0/problemdetermination/wasv7_gcmvoverview/player.html IBM Corporation

34 Questions? ` The presenters of this session will be in Meet The Developers lab. Please stop by to further explore any aspects of the technical details in this presentation. Please inquire about our schedule if we are not present when you arrive IBM Corporation

35 Legal disclaimer IBM Corporation All Rights Reserved. The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others IBM Corporation