White paper Abstract This white paper describes how to use load balancing in a scaled-out EMC ViPR SRM deployment to avoid performance bottlenecks that could impact data collection and report generation. October 2015
Copyright 2015 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. VMware is a registered trademark of VMware, Inc. in the United States and/or other jurisdictions. All other trademarks used herein are the property of their respective owners. Part Number H14610 2
Table of Contents Executive summary... 4 Audience... 4 Purpose... 4 Concepts... 4 Arbiter... 4 Load Balancer Connector... 5 Update checks... 5 Example... 5 Load Balancer architecture... 6 Prerequisites... 9 vapp consideration... 9 Installation... 9 Additional configuration tasks... 13 Removing access restriction for localhost... 13 Adding additional backends... 13 Reconfiguring SolutionPacks... 14 Changing the balancing key... 14 Troubleshooting... 16 Check the instance name... 16 Check if services are running... 16 Check log files... 17 Balancing lists... 17 Check communication ports... 17 Summary... 17 3
Executive summary As new devices are added to EMC ViPR SRM, the overall load on the system will increase with new metrics being pushed into the databases. This is when a Load Balancer comes into play. Similar in purpose to a network level load balancer, ViPR SRM s Load Balancer is a component that allows metrics from various collectors to be spread across multiple Backends based on decisions from an Arbiter. This allows you to scale out a current ViPR SRM deployment, avoids performance bottlenecks that could impact report generation, and provides means for the system to grow as requirements evolve. Naturally, funneling all of the metrics through a single component is likely to become a performance bottleneck. To avoid this scenario, another component was designed: the Load Balancer Connector (LBC). With the Load Balancer Connector, it is possible to offload the burden on the Arbiter by sharing a global destination table of where a given metric should go with all of the LBCs. Audience This article is for ViPR SRM installers, administrators or anyone who manages the ViPR SRM application. Purpose After reading this article, you understand load balancing concepts used with ViPR SRM along with specific procedures and recommendations for installation. Concepts Arbiter There is a need for only one Arbiter, as he is the one who decides to what backend where the metrics should go. The case when the Arbiter is out of service (i.e.: Virtual Machine downtime), the only consequence will be that metrics from devices newer to the Arbiter, will be retained locally on their respective collectors until the Arbiter is back to service those requests. Because all of the LBCs have a copy of the devices routing table, all of the previously learned devices will work as if nothing has occurred. Note: The Arbiter is not a single point of failure. 4
Load Balancer Connector The LBC is responsible for receiving the metrics from all of the collectors that point to it and to forward these metrics to the correct backend. Every time a new metric is received by the LBC, it checks on a local routing table, kept in a file, which backend that metric should be routed to. Whenever a device is seen for the first time, as the LBC will not have that information on its routing table file, it will proxy that metric to the Arbiter. The Arbiter then will, based on a routing algorithm, place the metric on a Backend, update its own balancing list file and send this update to all of its known LBCs. Note: Only one LBC should be installed once per VM/physical host on which the collection of data occurs. For example, multiple collector-managers will share one LBC instance on any given host. Update checks By default, the LBC will check for an update from the Arbiter every 60s (update-checktime) as shown in the balancer-connector.xml under /opt/apg/collecting/load- Balancer/Load-Balancer. Example <?xml version="1.0" encoding="utf-8"?> <balancing-configuration xmlns="http://www.watch4net.com/load-balancer" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://www.watch4net.com/load-balancer loadbalancerconfiguration.xsd "> <persistence> <file-persistence folder="data" /> </persistence> <balancer-connector> <arbiter-balancing-service validate-ssl="false"> <username>admin</username> <password>changeme</password> <url>https://localhost:48443/collecting/collector- Manager/INSTANCE/NAME/?wsdl</url> </arbiter-balancing-service> <update-check-time>60s</update-check-time> <arbiter-connector> <filter-configurations> <location>../../failover-filter/instance</location> <configurations>conf/failoverarbiter.xml</configurations> </filter-configurations> <connector-configuration type="socket-connector"> <location>../../collector-manager/instance</location> <configurations>conf/socket-arbiter.xml</configurations> </connector-configuration> </arbiter-connector> </balancer-connector> </balancing-configuration> 5
Load Balancer architecture New metrics arrive at the Load Balancer Connector from one or more Collector. As this is the first time the LBC sees these new metrics, it will forward them to the server where the Arbiter is installed. The Arbiter, in turn, will place these metrics on the Backends that are currently known to it. 6
The Arbiter shares its recently created balancing list with all of the LBCs registered with it. 7
With knowledge of the balancing list, the LBCs will be able to send the metrics to the proper Backends without having to consult the Arbiter. The LBCs will regularly ask for an updated version of the balancing list from the Arbiter, by default every 60 seconds. 8
Prerequisites EMC M&R 6.3 or higher Certify that the ports 2020 (Loadbalancer components) and 48443 (Webservice-Gateway) are not blocked between the collectors and the Arbiter. Procedures vapp consideration As of ViPR SRM 3.5, the LBC is already installed on the Collector Virtual Machine when deploying the vapp that bundles 4 VMs. If you are running this configuration, the following procedure is not necessary. Installation 1. Navigate to Centralized-Management > Physical Overview > <host > > Modules. Select the Collector host on which you want to install the Load Balancer connector module. 2. Click Install. The Packages Installation page displays. 9
3. In the Categories list, click Block. 4. In the Packages list select load-balancer-connector. 5. Click Launch and install the four modules (Collector-Manager, FailOver-Filter, LoadBalancer and the load-balancer-connector). 10
6. On the Load Balancer Connector, the following table describes the expected values: Field Web-Service gateway hostname or IP address [localhost] Web-Service port number [48443] Web-Service username [admin] Web-Service password [ ] Web-Service Category [Collecting] Web-Service Module [Collector-Manager] Web-Service Instance [Load-Balancer] Socket Collector port [2020] Arbiter hostname or IP address Value Primary Backend IP (vapp deployment) Default Default Default Default Default This needs to match the Arbiter Web-Service Instance. By default it is "Load-Balancer" Default Primary Backend IP (vapp deployment) 11
12
Additional configuration tasks Removing access restriction for localhost Remove the access restriction for localhost by commenting the last line in APG- WS.xml: <!-- Restrict access to localhost only <Valve classname="org.apache.catalina.valves.remoteaddrvalve" allow="127.0.0.1" /> --> Adding additional backends Each time a new Backend is added, the Arbiter must be informed about it. This can be done by executing, on the Arbiter machine, the reconstruction script (load-balancerreconstruction.sh) found under the bin directory of the Load Balancer collecting instance. PRIMARY_BE:/opt/APG/Collecting/Load-Balancer/Load-Balancer # bin/load-balancerreconstruction.sh conf/arbiter.xml conf/reconstruction.xml Retrieving keys for: device Connecting to end point Backend0 at jdbc:mysql:// PRIMARY_BACKEND:53306/apg Generated 4 balancing keys! Connecting to end point Backend1 at jdbc:mysql://additional_backend:53306/apg1 Generated 1 balancing keys! Connecting to end point Backend2 at jdbc:mysql://additional_backend:53306/apg2 Generated 12 balancing keys! Connecting to end point Backend3 at jdbc:mysql://additional_backend:53306/apg3 Generated 4 balancing keys! Connecting to end point Backend4 at jdbc:mysql://additional_backend:53306/apg4 Generated 7 balancing keys! Reconstruction is completed for the Arbiter. In order to be restarted safely, make sure that the Balancer Connectors tmp folders have been cleaned. Moreover, since the overwritten persisted keys were at version 1, the persisted keys on each Balancer Connectors MUST be cleaned manually to prevent unexpected behavior! PRIMARY_BE:/opt/APG/Collecting/Load-Balancer/Load-Balancer # Reconstruction should also take place whenever a database operation takes place (e.g., a database split). In order to use the reconstruction script you must first stop the Arbiter and then run the script. Reconstruction can also be used when the Arbiter is broken or when a new database is included. 13
Reconfiguring SolutionPacks With the new LBC in place, you can now reconfigure the SolutionPacks to make use of it. After being reconfigured, they will send their data through the LBC instead of aiming directly to the Backend. 1. Select a SolutionPack and then select its Data collection component. 2. Reconfigure the SolutionPack by changing its Data collection parameters so that it uses the new LBC, listening on port 2020 on the localhost. 3. Click Reconfigure. Repeat this process for all of the SolutionPacks on this host. If the LBC configuration is established before SolutionPacks are installed, it can be used for all SolutionPacks you subsequently install. Changing the balancing key By default, the Arbiter will take the property device as its balancing key. It is possible to modify this for other values but in most cases this will turn out to be a bad decision causing performance issues. NOTE: Changes to this file are not preserved during an upgrade or reconfiguration of the SolutionPack. For more information please refer to the APG-Load-Balancer.pdf document. At the Arbiter's main file (arbiter.xml), it is also possible to identify what Backends the Arbiter is currently aware of. They are listed under the container name "endpoints". <?xml version="1.0" encoding="utf-8"?> <!-- * Copyright (c) 2014, EMC Corporation. * All Rights Reserved. * This software contains the intellectual property of EMC Corporation * or is licensed to EMC Corporation from third parties. * Use of this software and the intellectual property contained therein * is expressly limited to the terms and conditions of the License * Agreement under which it is provided by or on behalf of EMC. --> 14
<!-- *********************************************************************************************** * This file has been auto-generated from SolutionPack code and should not be edited manually. * * Any manual changes in this file can potentially be lost. * * Edit this file only on formal recommendations from EMC. * *********************************************************************************************** --> <balancing-configuration xmlns="http://www.watch4net.com/load-balancer" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://www.watch4net.com/load-balancer loadbalancerconfiguration.xsd "> <persistence> <file-persistence folder="data" /> </persistence> <arbiter decision-manager="loadfactor"> <balancing-properties property="device" allow-missing-property="false" /> <end-points name="backend0" > <load-factor-service validate-ssl="false"> <username>admin</username> <password>{0192b122e0f172ae13ac8d79b32c5f909f81fd336746d474e570148a36422b6d614483d457593269c91d5d7ecf B732E7}</password> <url>https://primary_be:48443/backends/apg-backend/default?wsdl</url> </load-factor-service> <filter-configurations> <location>../../failover-filter/load-balancer</location> <configurations>conf/failover-backend-0.xml</configurations> </filter-configurations> <connector-configuration type="socket-connector"> <location>../../collector-manager/load-balancer</location> <configurations>conf/socket-backend-0.xml</configurations> </connector-configuration> </end-points> <end-points name="backend1" > <load-factor-service validate-ssl="false"> <username>admin</username> <password>{b44519f3432ad422202e0337191a88192a5655e2c49d72f8d6f29ac49c74fce41c469cba038db75e9a5b54189a F56E5A}</password> <url>https://additional_be:48443/backends/apg-backend/apg1?wsdl</url> </load-factor-service> <filter-configurations> <location>../../failover-filter/load-balancer</location> <configurations>conf/failover-backend-1.xml</configurations> </filter-configurations> <connector-configuration type="socket-connector"> <location>../../collector-manager/load-balancer</location> <configurations>conf/socket-backend-1.xml</configurations> </connector-configuration> </end-points> <end-points name="backend2" > <load-factor-service validate-ssl="false"> <username>admin</username> <password>{88a11c25a67fdaeed51b5bd9017c1cf416b2503421c87c3423c7c22a84126809d1362e3172d28b4b0dda0b8ef5 9621DF}</password> <url>https://additional_be:48443/backends/apg-backend/apg2?wsdl</url> </load-factor-service> <filter-configurations> <location>../../failover-filter/load-balancer</location> <configurations>conf/failover-backend-2.xml</configurations> </filter-configurations> <connector-configuration type="socket-connector"> <location>../../collector-manager/load-balancer</location> <configurations>conf/socket-backend-2.xml</configurations> </connector-configuration> </end-points> <end-points name="backend3" > <load-factor-service validate-ssl="false"> <username>admin</username> <password>{e5773b2519f8f68d4d84f66a5d1a5bc27b6477349e4f5057bb586f99d8742f3fae24269b72633fbd68ce292a66 15
8052D5}</password> <url>https://additional_be:48443/backends/apg-backend/apg3?wsdl</url> </load-factor-service> <filter-configurations> <location>../../failover-filter/load-balancer</location> <configurations>conf/failover-backend-3.xml</configurations> </filter-configurations> <connector-configuration type="socket-connector"> <location>../../collector-manager/load-balancer</location> <configurations>conf/socket-backend-3.xml</configurations> </connector-configuration> </end-points> <end-points name="backend4" > <load-factor-service validate-ssl="false"> <username>admin</username> <password>{b66fe53a9c8587c455c6830d666899af287ca92b1b32211ea31a1afee306ed52ca5ffdcf26e791df5f07f44c55 546E7B}</password> <url>https://additional_be:48443/backends/apg-backend/apg4?wsdl</url> </load-factor-service> <filter-configurations> <location>../../failover-filter/load-balancer</location> <configurations>conf/failover-backend-4.xml</configurations> </filter-configurations> <connector-configuration type="socket-connector"> <location>../../collector-manager/load-balancer</location> <configurations>conf/socket-backend-4.xml</configurations> </connector-configuration> </end-points> </arbiter> </balancing-configuration> Troubleshooting Check the instance name Check the instance name. The instance name of the WebService URL depends on the instance name of the actual Backend on the remote host. Check if services are running Check if services are running on the collectors as well as the Arbiter server. For example: Collector:/opt/APG/bin #./manage-modules.sh service status all * Checking 'topology-mapping-service Default'... [ running ] * Checking 'webservice-gateway Default'... [ running ] * Checking 'collector-manager Generic-RSC'... [ running ] * Checking 'collector-manager Generic-SNMP'... [ running ] * Checking 'collector-manager Load-Balancer'... [ running ] * Checking 'collector-manager emc-watch4net-health'... [ running ] * Checking 'script-engine Default'... [ running ] Collector:/opt/APG/bin # 16
Check log files On the collectors where the LBC is configured, check for communication issues usin the log files located at: /opt/apg/collecting/collecting/load-balancer/logs If there is indeed a communication problem between the LBC and either the Arbiter of one of the Backends, the FailOver Filter will start and the logs will contain entries pointing to that event. Balancing lists The balancing lists are clear text files that can be found under /opt/apg/collecting/load-balancer/load-balancer/data/ and are named as Backend0_1, Backend2_1, and so on. Check communication ports Check the status of the communication ports (2020 and 48443). Summary Using the information in this article, you should be able to configure load balancing and enable SolutionPacks to take advantage of the configuration. You also have information to help troubleshoot load balancer issues you may encounter. 17