Log Insight Manager Deployment Guide VERSION: 3.0 UPDATED: OCTOBER 2015
Copyright Notices Copyright 2002-2015 KEMP Technologies, Inc.. All rights reserved.. KEMP Technologies and the KEMP Technologies logo are registered trademarks of KEMP Technologies, Inc.. KEMP Technologies, Inc. reserves all ownership rights for the LoadMaster product line including software and documentation. The use of the LoadMaster Exchange appliance is subject to the license agreement. Information in this guide may be modified at any time without prior notice. Microsoft Windows is a registered trademarks of Microsoft Corporation in the United States and other countries. All other trademarks and service marks are the property of their respective owners. Limitations: This document and all of its contents are provided as-is. KEMP Technologies has made efforts to ensure that the information presented herein are correct, but makes no warranty, express or implied, about the accuracy of this information. If any material errors or inaccuracies should occur in this document, KEMP Technologies will, if feasible, furnish appropriate correctional notices which Users will accept as the sole and exclusive remedy at law or in equity. Users of the information in this document acknowledge that KEMP Technologies cannot be held liable for any loss, injury or damage of any kind, present or prospective, including without limitation any direct, special, incidental or consequential damages (including without limitation lost profits and loss of damage to goodwill) whether suffered by recipient or third party or from any action or inaction whether or not negligent, in the compiling or in delivering or communicating or publishing this document. Any Internet Protocol (IP) addresses, phone numbers or other data that may resemble actual contact information used in this document are not intended to be actual addresses, phone numbers or contact information. Any examples, command display output, network topology diagrams, and other figures included in this document are shown for illustrative purposes only. Any use of actual addressing or contact information in illustrative content is unintentional and coincidental. Portions of this software are; copyright (c) 2004-2006 Frank Denis. All rights reserved; copyright (c) 2002 Michael Shalayeff. All rights reserved; copyright (c) 2003 Ryan McBride. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE ABOVE COPYRIGHT HOLDERS ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE ABOVE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of the above copyright holders.. Portions of the LoadMaster software are copyright (C) 1989, 1991 Free Software Foundation, Inc. -51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA- and KEMP Technologies Inc. is in full compliance of the GNU license requirements, Version 2, June 1991. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 2
Portions of this software are Copyright (C) 1988, Regents of the University of California. All rights reserved. Redistribution and use in source and binary forms are permitted provided that the above copyright notice and this paragraph are duplicated in all such forms and that any documentation, advertising materials, and other materials related to such distribution and use acknowledge that the software was developed by the University of California, Berkeley. The name of the University may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. Portions of this software are Copyright (C) 1998, Massachusetts Institute of Technology Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. Portions of this software are Copyright (C) 1995-2004, Jean-loup Gailly and Mark Adler This software is provided 'as-is', without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this software. Permission is granted to anyone to use this software for any purpose, including commercial applications, and to alter it and redistribute it freely, subject to the following restrictions: 1. The origin of this software must not be misrepresented; you must not claim that you wrote the original software. If you use this software in a product, an acknowledgment in the product documentation would be appreciated but is not required. 2. Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software. 3. This notice may not be removed or altered from any source distribution. Portions of this software are Copyright (C) 2003, Internet Systems Consortium Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. Used, under license, U.S. Patent Nos. 6,473,802, 6,374,300, 8,392,563, 8,103,770, 7,831,712, 7,606,912, 7,346,695, 7,287,084 and 6,970,933. Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 3
Table of Contents 1 Introduction... 5 1.1 Document Purpose... 6 1.2 Intended Audience... 6 2... 7 2.1 Configure Log Insight Message Split Interval... 7 2.2 Create the TCP Syslog Virtual Service... 8 2.3 Create the UDP Syslog Virtual Service... 9 2.4 Create the SSL Syslog Virtual Service... 11 2.5 Log Insight API Ingest Service... 13 2.5.1 Create the Log Split Content Rule... 14 2.5.2 Create the API Ingest Virtual Service... 14 References... 17 Document History... 18 Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 4
Introduction 1 Introduction VMware vcenter Log Insight delivers real-time log management and log analysis with machine learning-based Intelligent Grouping, high performance search and better troubleshooting across physical, virtual and cloud environments. Figure 1-1: Log Insight Manager and the LoadMaster The flow of traffic in the above diagram is as follows: 1. The syslog clients create logs 2. The syslog clients then send the messages to the Virtual IP address on the LoadMaster 3. The LoadMaster distributes these messages to the Log Insight nodes Log Insight supports receipt and ingestion of syslog messages that are sent over UDP, TCP, TCP with SSL encryption and via API. The LoadMaster provides specialized Log Insight-aware services to optimize high availability and scalability of Log Insight deployments. Users can then perform deep analytics, discovery and search of the ingested data to get an enhanced operational view of their environment. Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 5
Introduction An inherent challenge that arises when syslog messages are sent using methods other than UDP, is that clients will often open long-lived connections that are then used for large amounts of messages. With this behavior, even when a scaled out architecture and application load balancer are implemented, traffic is not distributed in a close-to-even fashion across the pool of available nodes. The LoadMaster offers a solution that allows messages to be parsed within a connection to allow a more even distribution across servers in a pool, as well as simplified scalability of Log Insight environments. 1.1 Document Purpose The purpose of this document is to explain how to configure the LoadMaster to optimize VMware Log Insight traffic flows. 1.2 Intended Audience This document is intended to be read by anyone who is interested in configuring the LoadMaster to optimize VMware Log Insight deployments Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 6
2 A number of Virtual Services will need to be created for the LoadMaster to work effectively with Log Insight. The services that will be used depends on the methods that will be used in the environment to send syslog messages to the Log Insight nodes. Refer to the sections below for detailed, step-by-step instructions. 2.1 Configure Log Insight Message Split Interval The Log Insight Split Interval value controls how many syslog messages should be sent to each server in the pool before moving to the next server. For example, if there are three Log Insight nodes and the Log Insight Message Split Interval is set to 1 - a single message will be sent to server A, and then to server B and then server C before again distributing a message to server A. To set the Log Insight Split Interval, follow the steps below: 1. In the main menu of the WUI, go to System Configuration > Miscellaneous Options > L7 Configuration. Figure 2-1: Log Insight Message Split Interval 2. Set the Log Insight Message Split Interval. The default value is 10. The range is 1-100. Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 7
2.2 Create the TCP Syslog Virtual Service A TCP syslog Virtual Service must be created if clients will send syslog messages to Log Insight over TCP. To do this, follow the steps below: 1. In the main menu of the LoadMaster WUI, select Virtual Services > Add New. Figure 2-2: Virtual Service parameters 2. Enter a valid Virtual Address. 3. Enter 514 as the Port. 4. Enter a recognizable Service Name, for example Log Insight TCP. 5. Click Add this Virtual Service. The Service Type (in the Basic Properties section) of the 514 Virtual Service will be set to Log Insight by default. 6. Expand the Standard Options section. Figure 2-3: round robin 7. Select round robin as the Scheduling Method. Round robin is typically best to accomplish desired behavior of even traffic distribution. Least connection will result in an uneven distribution for syslog over TCP, especially when there is a low number of connections. If the Scheduling Method is set to least connection and there are a low number of connections, the Log Insight Split Interval (see below) will not behave as expected. 8. Expand the Real Servers section. Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 8
9. Enter 514 as the health check port. 10. Click Add New. Figure 2-4: Real Servers section 11. Enter the Real Server Address. 12. Click Add This Real Server. Figure 2-5: Real Server Parameters 2.3 Create the UDP Syslog Virtual Service A UDP Syslog Virtual Service must be created if clients will send syslog messages to Log Insight over UDP. To do this, follow the steps below: 1. In the main menu of the LoadMaster WUI, select Virtual Services > Add New. Figure 2-6: Virtual Service parameters 2. Enter a valid Virtual Address. 3. Enter 514 as the Port. 4. Enter a recognizable Service Name, for example Log Insight UDP. 5. Select udp as the Protocol. 6. Click Add this Virtual Service. 7. Expand the Standard Options section. Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 9
Figure 2-7: Standard Options 8. Select the Force L7 check box. 9. Ensure that the Transparency check box is selected. This allows the client s IP address to be presented to the Log Insight servers. Depending upon your network topology, transparency may not be supported. If this is the case, you can safely disable this Transparency option and the source IP presented to Log Insight will be that of the Virtual Service. The hostname will remain unchanged. 10. Enter a low Idle Connection Timeout value and click Set Idle Timeout. A value of 1 will typically result in the best performance for the Log Insight UDP service. 11. Expand the Real Servers section. Figure 2-8: Real Servers section 12. Ensure that ICMP Ping is selected in the Real Server Check Parameters drop-down list. 13. Click Add New. 14. Enter the Real Server Address. Figure 2-9: Real Server Parameters Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 10
15. Click Add This Real Server. 2.4 Create the SSL Syslog Virtual Service A SSL syslog Virtual Service must be created if clients will send syslog messages to Log Insight over TCP. To do this, follow the steps below: 1. In the main menu of the LoadMaster WUI, select Virtual Services > Add New. Figure 2-10: Virtual Service parameters 2. Enter a valid Virtual Address. 3. Enter 1514 as the Port. 4. Enter a recognizable Service Name, for example Log Insight SSL. 5. Click Add this Virtual Service. 6. Select Log Insight as the Service Type. 7. Expand the SSL Properties section. Figure 2-11: Basic Properties Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 11
Figure 2-12: SSL Properties 8. Select Enabled. 9. Click OK. 10. Click Manage Certificates. Figure 2-13: Import Certificate 11. Click Import Certificate. Figure 2-14: Certificate details 12. Click the first Choose File button. 13. Browse to and select the relevant certificate file. 14. If needed, upload a Key File and enter the Pass Phrase. 15. Enter a name in the Certificate Identifier text box. 16. Click Save. Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 12
Figure 2-15: Assign the certificate 17. Select the relevant Virtual Service from the Available VSs box. 18. Click the right arrow button to assign the Virtual Service to the certificate. 19. Click Save Changes. 20. Expand the Real Servers section. Figure 2-16: Real Servers section 21. Ensure that TCP Connection Only is selected in the Real Server Check Parameters dropdown list. 22. Click Add New. 23. Enter the Real Server Address. 24. Enter 514 as the Port. 25. Click Add This Real Server. 26. Add any other Real Servers as needed. Figure 2-17: Real Server parameters 2.5 Log Insight API Ingest Service If HTTP POST requests will be used to programmatically send log information to the Log Insight cluster, a Log Split content rule is required and an accompanying Virtual Service must be created. Content rules interrogate incoming client connections and make decisions as well as header modification based on the contents of the requests. Follow the steps in the two sections Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 13
below for instructions on how to do this. This rule will ensure even distribution of messages across the cluster of Log Insight nodes when the API Ingest Service is utilized. 2.5.1 Create the Log Split Content Rule A Log Split content rule is required to minimize lumpiness and accomplish a more even distribution of messages that are posted. To create the content rule, follow the steps below: 1. In the main menu of the LoadMaster WUI, select Rules & Checking > Content Rules. 2. Click Create New. Figure 2-18: Content Rule 3. Enter a recognizable Rule Name, for example LogInsightAPI. 4. Select Replace Header as the Rule Type. 5. Enter Connection as the Header Field. 6. Enter keep-alive as the Match String. 7. Enter close as the Value of Header Field to be replaced. 8. Click Create Rule. For more information, refer to the Feature Description, Content Rules. 2.5.2 Create the API Ingest Virtual Service Now, an API ingest Virtual Service must be created. To do this, follow the steps below: 1. In the main menu, select Virtual Services > Add New. 2. Enter a valid Virtual Address. Figure 2-19: Virtual Service parameters Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 14
3. Enter 9000 as the Port. 4. Enter a recognizable Service Name, for example Log Insight API. 5. Click Add this Virtual Service. 6. Select HTTP/HTTPS as the Service Type. 7. Expand the Standard Options section. Figure 2-20: Basic Properties Figure 2-21: Standard Options 8. Ensure that the Transparency check box is selected. 9. This allows the client s IP address to be presented to the Log Insight servers. Depending upon your network topology, transparency may not be supported. If this is the case, you can safely disable this Transparency option and the source IP presented to Log Insight will be that of the Virtual Service. The hostname will remain unchanged. 10. Expand the Real Servers section. Figure 2-22: Real Servers section 11. Ensure that TCP Connection Only is selected in the Real Server Check Parameters dropdown list. 12. Click Add New. Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 15
Figure 2-23: Real Server parameters 13. Enter the Real Server Address. 14. Click Add This Real Server. 15. Click OK. 16. Add any other Real Servers as needed. 17. Click Back. 18. Expand the Advanced Properties section. 19. Click Enable. 20. Click Show Header Rules. Figure 2-24: Advanced Properties Figure 2-25: Assign rule 21. In the Request Rules section, select the relevant rule and click Add. Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 16
References References Unless otherwise specified, the following documents can be found at http://kemptechnologies.com/loadmaster-documentation. Feature Description, Content Rules Web User Interface, Configuration Guide Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 17
Document History Document History Date Change Reason for Change Version Resp. Sep 2014 Initial draft First draft of document 1.0 LB Nov 2014 Minor update Defects resolved 1.1 LB Jan 2015 Release updates Updates for 7.1-24 release 1.2 LB June 2015 Release updates Updates for 7.1-28 release 1.3 LB Oct 2015 Release updates Updates for 7.1-30 release 3.0 LB Copyright 2002-2015 KEMP Technologies, Inc. All Rights Reserved. 18