EMC KAZEON E-DISCOVERY: PERFORMANCE TUNING GUIDELINES AND BEST PRACTICES

Transcription

1 EMC KAZEON E-DISCOVERY: PERFORMANCE TUNING GUIDELINES AND BEST PRACTICES ABSTRACT EMC Kazeon performance is significantly dependent on the I/O operations performed and also on the network bandwidth. This white paper explains various guidelines and best practices which can help to improve the user experience and system performance significantly. March, 2015 EMC WHITE PAPER

2 To learn more about how EMC products, services, and solutions can help solve your business and IT challenges, contact your local representative or authorized reseller, visit or explore and compare products in the EMC Store Copyright 2014 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. VMware and <insert other VMware marks in alphabetical order; remove sentence if no VMware marks needed. Remove highlight and brackets> are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other trademarks used herein are the property of their respective owners. Part Number H

3 TABLE OF CONTENTS EXECUTIVE SUMMARY... 4 AUDIENCE... 4 INTRODUCTION... 4 GUIDELINES AND BEST PRACTICES... 4 Online vs offline processing... 5 Running parallel jobs... 5 Adding nodes to the cluster to improve performance... 5 Mapping kazfs, datafs and jobs... 5 Network related parameters and configuration... 6 Configuring dedicated channel for Kazeon to access storage... 6 Monitoring available local disk space, remote disk space, CPU usage and memory... 7 Periodic system configuration backup... 7 Periodic password change exception... 7 Monitoring: DB maintenance, vacuum, cluster restart after long use... 7 Throughput of kazfs and datafs using dd and testfs utility... 7 No 3 rd -party tools, scripts, security scans, or anti-virus... 8 Keep file system offline when not in use. Bring online when required... 8 Basic crawl flag to determine the object type (doctypemode)... 8 Process heavier objects using staging for exchange and CSVC repositories... 8 Clearing unused/stale rulesets from LDAP using LDAP cleanup tool... 8 On-demand search index optimization tool... 8 Use of Kazeon side vs agent side filters... 9 Object copy vs object collection... 9 CONCLUSION

4 EXECUTIVE SUMMARY EMC Kazeon performance is significantly dependent on the I/O operations performed and also the network bandwidth. This white paper explains various guidelines and best practices which will help you to tune up and configure the Kazeon system for better performance. The paper also talks about how to organize the data, metadata repositories and the jobs to harness the potential of the system. AUDIENCE This white paper is intended for engineering, functional and support teams who work related to enterprise search and/or ediscovery using Kazeon. It is assumed that the reader of this document has hands-on experience of EMC Kazeon ediscovery. INTRODUCTION EMC Kazeon server performance is significantly impacted by the I/O operations performed by the system. The Kazeon servers perform significant amount of I/O operation for crawling data, copying data, moving data, deleting data, downloading data, indexing data, querying data and searching Data. The I/O operations performed by Kazeon servers make the I/O channel a bottleneck for the overall performance of the Kazeon. GUIDELINES AND BEST PRACTICES Online vs offline processing Running parallel jobs Adding nodes to the cluster to improve performance Mapping kazfs, datafs and jobs Network related parameters and configuration Configuring dedicated channel for Kazeon to access storage Monitoring available local disk space, remote disk space, CPU usage and memory Periodic system configuration backup Periodic password change exception Monitoring: DB maintenance, vacuum, cluster restart after long use Throughput of kazfs and datafs using dd and testfs utilities No 3 rd -party tools, scripts, security scans, or anti-virus Keep file system offline when not in use. Bring online when required Basic crawl flag to determine the object type (doctypemode) Process heavier objects using staging for exchange and CSVC repositories Clearing unused/stale rulesets from LDAP using LDAP cleanup tool On-demand search index optimization tool Use of Kazeon side vs agent side filters Object copy vs object collection 4

5 Online vs offline processing Broadly, Kazeon performs two kinds of jobs 1- On-line processing (interactive) 2- Offline processing (non-interactive) During online processing the users (admin, supervisor, legaladmin, reviewers, etc.) logon to the Kazeon system UI and perform system interactive tasks. Offline processing mostly comprises of non-interactive admin operations like crawl, collection, copy, etc., which does not require manual intervention once the service is triggered. Kazeon system is designed to not to differentiate between online and offline processing. So it makes it obvious and sensible to give preference to the online processing as this involves user time. So it is always advisable to plan your offline jobs and schedule them in such a way that they do not compete for system resources during your online hours. Scheduling your offline jobs during non-peak hours, weekends and holidays helps you to keep the system resources available for online processing during peak hours. Running parallel jobs Kazeon 4.8 (64-bit) is designed to run 8 concurrent deep crawl jobs or 16 concurrent basic crawl jobs to achieve the maximum throughput. Running more than recommended jobs will result in thrashing and degradation of the system performance. It is not recommended to go beyond the recommended number. It is also very important to understand that running just 1 Job will not produce same performance result when compared to multiple jobs. The system performance is computed based on the sum of all the jobs performance running concurrently. So to gain the maximum processing throughput, split your single Job in such a way that it can be run as multiple jobs to divide the work and achieve the peak performance. Adding nodes to the cluster to improve performance A Kazeon cluster can have minimum of one node to maximum four nodes in a cluster. By adding additional nodes you can increase the overall performance and capacity of the cluster. It is very important to understand that the by adding additional node to the cluster does not adds up to the processing capacity linearly. It is observed that the performance adds up by approximately 0.5x of single node cluster performance. This behavior is due to the network latency involved in the communication among the nodes in the multi-node cluster over network. Some of the Kazeon processes (index optimization process among others) are sequential in nature and cannot be run in parallel on multiple nodes. You can expect a better performance by adding more nodes but not in a linear fashion. By adding additional node you not only add up to the processing speed but also to the processing capacity. Mapping kazfs, datafs and jobs Kazeon processes are highly I/O intensive as it has to perform lot of reading/writing from datafs and kazfs for all of its supported operations (crawl, collection, move, delete, tagging etc.). Although, we have OS based disk caching and file I/O caching support in place but the disk caching does not help the processing much as all of the disk file operations are one time. Disk and file caching helps only when we access the file for multiple operations. In most of the cases, it is observed that the I/O channel is the root cause for the bottleneck for overall performance of the Kazeon. So to achieve the maximum throughput by running multiple jobs, it is necessary to arrange the kazfs, datafs and jobs in such a way so that they do not block each other and do not compete for I/O. It is not recommended to have multiple datafs mapped to a single kazfs or multiple jobs running out of single datafs concurrently. This setup will not only make the I/O channel of datafs or kazfs, a bottleneck but also it will become huge in size for search and related operations. It is always recommended to have : kazfs and datafs in ratio of 1:2. That is 2 datafs mapped to 1 kazfs. In general, for n kazfs there should not be more than 2xn datafs. Moreover, associated metadata and data repositories should not be on the same server, otherwise performance issues arise. The minimum latency for sustained throughput on any metadata or data repository should be at least 60 MB/s. As far as possible try to have datafs and kazfs on separate servers. The datafs and kazfs servers should be equipped with gigabit NIC interface and supporting network. Avoid using datafs/kazfs mounted on mixed-mode or multi-protocol environment as this add up to the performance drop. 5

6 Figure 1 below, shows recommended topology of the filesystems (data and metadata) and jobs on Kazeon to achieve maximum performance. You can have any number of datafs and kazfs as far as you are within prescribed limits as per the product documentation. But it is important to make sure, not more than 2 jobs should be writing into a single kazfs simultaneously. It is also important to make sure that you do not spawn too many jobs originating from a single source repository concurrently. Figure 1: Recommended datafs-kazfs mapping By running only one basic crawl job will not be able to harness the potential of hardware to achieve the required performance. As per best practices recommendation, the setup should have multiple kazfs load balancing the datafs as depicted above. It is also recommended to create multiple jobs per datafs by logically dividing the data in a datafs and assigning it to each job. (eg: Create jobs at sub-folder levels within datafs instead of root level). Network related parameters and configuration It is not recommended to have more than 2 hops between the Kazeon server and the datafs or kazfs server. Each hop s MTU should be well aligned with the Kazeon server s MTU and the storage server s MTU to avoid data packet drops and losses due to packet drops. It is recommended to have Kazeon and the filesystem (datafs and kazfs) servers in same subnet to reduce network latency. By default, the MTU size of the Kazeon server is 1500 for eth1 and eth2; and for local loopback the MTU size is It is also very important to have low latency. The datafs and kazfs servers should be equipped with gigabit NIC interface and supporting network. Various Linux-based network tools can be used to determine above configurations eg. traceroute, ping, etc. Configuring dedicated channel for Kazeon to access storage Now a day s most of the advanced storage servers provide network load balancing of the client connection to the storage. The load balancer decides the connection to storage from the client and maps them to the NIC. Load balancers are designed to provide High availability but do not guarantee high bandwidth connection to the client. Load balancer goes by fair distribution policy where every client connection is treated fairly irrespective of the client demand for bandwidth. Hence, on a busy day every client can experience slowness because of load balancer. By bypassing the load balancer and getting a direct client connection to not so loaded NIC interface helps Kazeon to get higher bandwidth for I/O operation. Eg. Isilon uses SmartConnect Zone (Isilon Smart Connect IP) for load balancing client connections. 6

7 Monitoring available local disk space, remote disk space, CPU usage and memory Keeping system resources free and healthy gives the system enough space to breathe and run freely. Periodic checks on the local disk space help the system to utilize it for day-to-day system activities like logging, dumping core, downloading, unpacking of complex objects, parsing, LDAP caching, virtual memory, and various other admin activities related to report generation, preview of objects, generating health-check logs & grab diagnostics. Almost all of the activities take some disk space for operation. Keeping system free space in safe limits helps system run smoothly. Never add (non-system or user created) files to the IS1200 /tmp directory. Over-filling this directory can cause inexplicable errors and system slow-downs. Periodic system configuration backup It is recommended to perform periodic backups which should include system configs, LDAP, kazschema, critical metadata repositories, etc. which can help in easy system recovery in case of system failure or crash. Periodic password change exception Security policies that require passwords to expire and be changed on a periodic basis pose problems when Kazeon uses Active Directory for authentication or to mount CIFS shares. It is recommended to exclude the user account or identity, stored in identity vault, from this policy. It this is not feasible, the Kazeon administrator must change the passwords for the identities from Kazeon identity vault whenever there is password change in AD side. Monitoring: DB maintenance, vacuum, cluster restart after long use Letting system to run periodic auto DB maintenance helps the DB to recover and makes faster DB access. For auto DB maintenance and auto vacuum to run the system needs to be kept idle periodically. System does not schedule auto maintenance and auto vacuum process whenever system is busy and processing. Letting the system to be idle periodically make the system schedule the maintenance process and makes system stable and perform better. Over long continuous runs and log system up time system tends to leak resources and becomes dirty and unstable due to leaks. It is advisable to reboot the cluster and nodes periodically after a long uptime to reclaim the leaked system resources back. Note: You will experience system slowness while vacuuming and DB maintenance process are running, this is expected behavior. Throughput of kazfs and datafs using dd and testfs utility The jobs performance in Kazeon is largely dependent on the network throughput between Kazeon and the filesystem (data and metadata) servers. The minimum latency for sustained throughput on any metadata or data repository should be at least 60 MB/s. Hence, before setting up any repository and triggering any jobs, make sure that you have sufficient network bandwidth. To check the throughput, one can use Linux utility dd or Kazeon utility testfs Example: dd Utility: Write performance: dd if=/dev/zero of=<kazfs>/perf.rand bs=1g count=1 Read performance: dd if=<kazfs>/perf.rand of=/dev/zero bs=1g count=1 testfs Utility: testfs [-r numreadthreads] [-n numreads -t seconds] [-s filesize] [-c] [-S readsize] [-w numwritethreads] f filename [ d] testfs should be run with at least -t 60 to allow for variances in system, network, and server loads that can impact average performance. It should be run multiple times during the day, at all typical network load levels. For more details refer to Kazeon webadmin user guide. Note: It is always recommended to use testfs utility for throughput measurement as it does random I/O which is more close to Kazeon I/O usage as compared to dd which performs sequential I/O. 7

8 No 3 rd -party tools, scripts, security scans, or anti-virus It is not recommended to have any other application running on the Kazeon appliance other than native Kazeon services and applications. Running such 3 rd -party tools, anti-virus scans, port scans may result in system performance degradation and system failures. The system is expected to be running in intranet and should always be kept behind corporate firewalls. Kazeon system consumes huge amount of system resources during run and it rapidly downloads and opens lots of file for operation. Any anti-virus scan running on the system blocks the file operation and hiders the system performance significantly. We have also seen instances where Kazeon system going down because of security scan running on the open ports of the Kazeon system which are meant for inter-cluster communication. So it is always recommended to keep the system behind corporate firewall and free from third party applications and scans. Keep file system offline when not in use. Bring online when required It is recommended to keep the repositories offline when not required for long. Keeping the unused repositories offline helps the system to save system resources spent periodically on collecting repository health and other repository management related activities. Switching repositories between offline and online is not time consuming process. So whenever a repository is required, bring it up in no time and get going. This also helps you to maintain system dashboard clutter free. Basic crawl flag to determine the object type (doctypemode) The Basic crawl performance can be significantly improved by disabling the LDAP flag doctypemode. This flag when enabled forces the system to collect object type information along with other filesystem metadata. If an end-user is not interested in document type metadata information, she can switch off the doctypemode flag to gain additional performance. Note this flag only helps to boost the performance of basic crawl by bypassing the additional step of determining the object type information. Warning: This flag only helps to boost the performance of the basic crawl. No deep crawl job should be executed with this flag set to off, as this will result in performance degradation and undesirable results for the deep crawl jobs. The deep crawl jobs should always be executed when this flag is set to ON. Please refer to Kazeon web-admin user guide for more details. Process heavier objects using staging for exchange and CSVC repositories For processing bigger and heavier objects in Exchange or CSVC repositories, it is not advisable to increase the configuration parameter on the agent to process the objects. By increasing the object size parameter only overloads the regular IS1200 and Exchange/CSVC agent web-service channel, which in turn reduces the performance. So instead of increasing the object size parameter on the Exchange/CSVC agent, it is advisable to make use of download handler feature on the agent which queues up the heavier object on a separate channel for download. This keeps the regular queue unblocked for quickly processing lighter objects. Clearing unused/stale rulesets from LDAP using LDAP cleanup tool When you run a single step collection or an in-place processing service, Kazeon auto generates extraction and assignment rulesets based on the job parameters. These rulesets reside in the LDAP server and are never reused once the services are completed. These unused rulesets wastes space on LDAP server and hence may lead to slow response of the LDAP server. To clear the unused/stale rulesets, you can use Stale rulesets cleanup tool. The syntax to run the tool is: clean_ldap_rulesets [--dry-run] {--all --mm/dd/yyyy} For more details on the tool, please refer to Kazeon administrator user guide. On-demand search index optimization tool The search indices created by Lucene (the indexing engine used by EMC Kazeon Search) gets fragmented over the time resulting in low search performance. Lucene optimization is a defragmentation process and speeds up the search performance. The index optimization is a resource intensive process and hence if the auto-optimization is triggered through search workflow, users may experience performance issues. To remedy this, you can use the on-demand search index optimization tool, which provides the flexibility to optimize indexes on demand. The syntax to run the tool is: optimizer.sh {--optimize --dry-run} {--fsid {<comma separated datafs_id> ALL}} For more details on the tool, please refer to Kazeon 4.8 release notes. 8

9 Use of Kazeon side vs agent side filters Adding filters to the operation adds up to the significant performance degradation of a crawl. It is always recommended to use filters judiciously and avoid wherever possible. Using remote agent based filters will be faster when compared to filters on the IS1200 server side. Object copy vs object collection It may sound Copy and Collection process similar but they are designed for different purposes. Copy process is subset of collection process. Collection process includes copy operation and indexing process to make copied objects searchable. The additional indexing process makes collection process significantly slower than copy process. So next time initiating any collection process make sure you need indexing of the copied objects else make use of copy feature. CONCLUSION This white paper explained various guidelines and best practices which will help you to tune up and configure the Kazeon system for better performance. Ii also talked about how to organize the data, metadata repositories and the Jobs to harness the potential of the system. 9