Technical Note: TN184 Nextiva VMS Master Server Under VMware Note: This technical note replaces TN-6.X-166, 182, 184 Publication Date: April 2011 Product(s) Nextiva VMS VMware Exposure Level Type of Technical Note Customer Facing Product Integration Feature Description Applicable Software Package(s) Nextiva VMS 6.2 RU1 and later VMware ESXi Description This technical note describes the recommendations for deploying the VMware Solutions with the Nextiva Master Server, either for reducing the IT footprint or for Fault Tolerance (FT) and High Availability (HA) deployments. It also offers information on troubleshooting this solution for performance and ready time. Unauthorized use, duplication, or modification of this document in whole or in part without the written consent of Verint Systems Inc. is strictly prohibited. By providing this document, Verint Systems Inc. is not making any representations regarding the correctness or completeness of its contents and reserves the right to alter this document at any time without notice. Features listed in this document are subject to change. Please contact Verint for current product features and specifications. All marks referenced herein with the or TM symbol are registered trademarks or trademarks of Verint Systems Inc. or its subsidiaries. All rights reserved. All other marks are trademarks of their respective owners. 2012 Verint Video Intelligence Solutions. All Rights Reserved Worldwide.
Overview Today s organizations need robust and resilient video management systems to ensure that video is available at all times. To achieve this goal, they add Fault Tolerance (FT) and High Availability (HA) to their solutions. FT and HA solutions protect against unplanned downtime from common failures, such as network and storage interface failures or server failures. Fault Tolerance (FT) refers to the capacity of a system to ensure continuous processing even if one of its components fails. High Availability (HA) refers to the property of a multiprocessing system to recover quickly from a failure while processing continues. In FT or HA solutions, specialized software automatically puts standby services online. Verint teams up with EMC to add VMware Fault Tolerance and High Available to provide failover functionality to Nextiva VMS. If any hardware component failure is detected on the primary server, the Nextiva Master Server is fully transitioned to another ESXi server. The VMware FT environment provides a non-disruptive transition during failover with no data loss. Under the HA environment, however, the virtual machine reboots on the secondary server. The amount of time it takes for a Nextiva VMS system to fully recover depends on the size of the system and can take up to five minutes. This technical note offers guidelines for deploying Nextiva VMS in a VMWare environment and troubleshooting VMWare for performance and ready time as described in the following sections: Prerequisites CPU, Memory, Network and Storage vcpu, Memory, Network and Storage Guidelines for Storage Requirements Typical Supported Scenarios Troubleshooting VMware Performance and Ready Time Prerequisites VMware ESXi Microsoft Windows Server 2003 R2 SP2 x86, Microsoft Windows Server 2008 SP2 x86- x64 (guest operating system running under ESXi), Microsoft Windows 2008 R2 A cluster configuration with a minimum of two ESXi hosts running. A shared storage (iscsi, SAS or FC) configured and recognized for all servers within the cluster. CPU, Memory, Network and Storage It is highly recommended that the servers, storage adaptors, and other hardware you are using in the integration are officially validated by VMware. For the list of supported hardware, see http://www.vmware.com/resources/compatibility/search.php Note: The recommendations for running Nextiva VMS under a VMware environment are different from those for running Nextiva VMS without VMware. Verint Video Intelligence Solutions 2 of 11
The following recommendations correspond to a deployment with a minimum load on the system and reflect therefore minimum requirements. Prerequisites VMware FT VMware HA CPU Dual-Core Xeon Dual-Core Xeon ( up to 700 cameras) Quad-Core Xeon (more than 700 cameras) Memory 4 GB of RAM 4 GB of RAM Network Storage 3 Gigabit Ethernet Ports (Nextiva VMS Network, VMotion Kernel, Fault Tolerance Kernel) Virtual Datastore on iscsi, SAS or FC Supported by VMware for the VM 3 Gigabit Ethernet Ports (Nextiva VMS Network, VMotion Kernel, Fault Tolerance Kernel) Virtual Datastore on iscsi, SAS or FC Supported by VMware for the VM Notes: Ensure that the same processor model is used on both the Primary and Secondary ESXi servers for the FT solution. Add hardware if more VMs than the Master Server will be running on the ESXi host., vcpu, Memory, Network and Storage The specifications for the virtual hardware are equivalent to the recommended specifications for the Master Server described in the Nextiva VMS Customer Furnished Equipment Guide. Note: Ensure that the virtual machine is not using more than one vcpu for the FT Solution this is a VMware design requirement. The following table lists the minimum requirements the virtual hardware It also offers recommended specifications for optimum performance. Prerequisites Master Server Memory Recommended 1 vcpu (FT) 2 GB 700 cameras 2 vcpu (HA) 2 GB 700 cameras 4 vcpu (HA) 4 GB + 700 cameras Memory 4 GB Additional 2 GB for every 2 VMs hosted on a single hardware platform Additional 4 GB for the IVAP server Network 1 Gigabit Ethernet Port for each pair of VMs 1 dedicated Gigabit Ethernet port for the ESXi Console 1 dedicated Gigabit Ethernet port per VM Storage 80 GB of un-partitioned disk space for each VM, or Direct attached, or SAN devices 4 Gbps FC SAN for Video Storage Verint Video Intelligence Solutions 3 of 11
Guidelines for Storage Requirements This section provides the minimum storage requirements for the Recorder Server and ESM Servers. Note: On Nextiva VMS, Video disks use only Raw Device Mapping (RDM) under ESXi. RDM offers better performance because it is a direct communication to the storage without creating a file system like VMFS. Fibre Channel SAN It is highly recommended to use a Fibre Channel SAN when ESXi is hosting any number of Nextiva Recorder or Nextiva ESM. The recommended minimum fibre channel speed is 4Gbps. iscsi SAN A typical iscsi offers less performance than Fibre Channel SAN because of the following reasons: iscsi link speed is normally limited to 1Gbps Ethernet speed vs Fibre Channel running at 4Gbps (10Gbps iscsi is not, at time of updating this document, broadly available). Traffic switching and routing may introduce significant latency or packet loss, impacting the effective storage throughput. iscsi Software Adaptor (initiator) increases the load in the CPU of the host server. Recommendations: Use Fibre Channel whenever possible Configure a dedicated vswitch for vmkernel (ensure not to share a vswitch between vmkernel port group and Virtual Machine port group). Configure multiple network adaptors in load balancing for the vswitch hosting the vmkernel. If recommended by the iscsi SAN vendor, implement Jumbo packet on your network. Recommended VM Configuration Configuration: Reserve 1200 Mhz for each VM (recommended but not required under ESXi 4.0 and ESXi 4.1) Runtime: To ensure the smooth execution of every VM, keep the average CPU utilization of the host below 55% of the total CPU usage. Verint Video Intelligence Solutions 4 of 11
Typical Supported Scenarios The following table illustrates the number of virtual machines that are supported per type of CPU and available memory. CPU Type Memory Number of VM Description Single Quad Core 4GB 2 1 Master server VM (up to 700 cameras) 1 Recorder server VM Dual Quad Core 12GB 2 2 Recorder server VM Dual Quad Core 12GB 2 1 Master server VM (up to 2,000 cameras) 1 Recorder server VM Quad Quad Core 24 GB 4 4 Recorder server VM Quad Quad Core 24 GB 4 1 Master server VM (up to 2,000 cameras) Support Statement 3 Recorder server VM Verint supports customers using virtualized environments; however: Should Verint suspect that the virtualization layer is the root cause of an incident; the customer will be required to contact the appropriate VMware support provider to resolve the VMware issue. Verint can ask a customer to reproduce a problem in a non-virtualized environment if Verint determines that the virtualization component can be the root cause of the problem. Verint Video Intelligence Solutions 5 of 11
Troubleshooting VMware Performance and Ready Time This section describes the steps to follow for a noticeably degraded VMware solution in a Nextiva VMS system. It will help you identify performance issues and overloading of the virtual machine s host. In order to perform these tasks, a minimum understanding of VMware knowledge is required. This includes the use of the VI Client and the ESXi host. This section covers guidelines and procedures on: Performance Monitoring Assessing Ready Time Status Performance Monitoring The VI Client helps you identify a problem on both the host and the virtual machine. There are two symbols to pay attention to: the Yellow Caution Alert and the Red Alert. These alerts indicate if there is a problem that must be either resolved or monitored. They are triggered when a threshold is surpassed with or without a condition. Examples include sending a warning if the CPU exceeds 85% for five minutes and sending an alert if the CPU attains 100% for five minutes. Some of the rules that can be modified are: Host Connection and Power State Host Hardware Power Status Host Storage Status Host CPU Usage Host Memory Usage Host Processor Status Host Hardware System Board Status Host Memory Status The alerts are defined within the VI Client by highlighting the Cluster, Host, or VM, then clicking on the Alarms tab in the right pane. The VI Client provides a powerful tool for monitoring the performance of the CPU, Memory, Disk, Network, System, and other components in real time. To monitor performance: 1. Open the VI Client. 2. Do one of the following: Highlight the ESXi host for a general real-time data capture Highlight any specific virtual machine for more accurate data. 3. Click on the Performance tab. The Real-Time Graph is displayed. Verint Video Intelligence Solutions 6 of 11
For troubleshooting purposes, the Real-Time Graph can be exported as an image or to an Excel file with all the data that is displayed data. This can help track multiple time slots in order to analyze if the CPU or memory are growing with time. See illustration below: Save Option for Excel Verint Video Intelligence Solutions 7 of 11
Another interesting feature is the Chart Options Users can specify a range of dates, objects and counters to be displayed on the chart. For example, choose the option Past Year with Object 1, 2 for CPU Core 1 and 2, and the counter Usage as an average. Then, export the results for further analysis. The Chart Options are illustrated in the following screen shot: Assessing Ready Time Status To ensure that the system is properly installed and configured, you need to confirm that the Ready Time Status of the virtual machines is appropriate. The definition of Ready Time is the time that a virtual machine waits in a ready-to-run state before being scheduled on a CPU. This section describes the following: Identifying CPU and Ready Time Applying the Prevention Method Verifying Total CPU Use Verint Video Intelligence Solutions 8 of 11
Identifying CPU and Ready Time Statistics You need a direct console connection to the ESXi host or a SSH connection using a tool such as Putty in order to identify ready time statistics. All the following can affect Ready Time: Busy ESXi console Sharing resources (Memory, CPU, Disk ) Large number of virtual machines compared to hardware availability Overall CPU utilization To identify Ready Time Statistics: 1. Type esxtop 2. Press the key c. 3. Look at the %RDY for the virtual machines and the console as illustrated below. The following parameters are displayed: Load Average: This is the use for all physical CPUs. It displays for one-, five-, and fifteen minutes intervals. PCPU: Percentage of the individual physical CPU CCPU: Percentage of Total CPU Time us: % User Time sy: % System Time id: % Idle Time wa:% Wait Time cs-sec: Context switch per second Verint Video Intelligence Solutions 9 of 11
Identifying a Bad Ready Time A bad Ready Time can result in a slow virtual machine or loss of communication causing network failure or bad data transfer. The lower the percentage, the better the Ready Time. A low percentage indicates that the virtual machine did not waste time waiting for the next CPU cycle. 0 to 5% is required 5 to 10% is acceptable but leaves room for improvement 10% + is not acceptable If the percentage is too high, it most probably means that the system is overloaded with virtual machines and/or sharing too many resources. Methods for preventing a bad Ready Time include: CPU Reservation Setting Affinity to protect the ESXi Console from Overload The total CPU use for the host should be under 55% Applying the Prevention Method The prevention method helps ensure that no virtual machine is using CPU processing from the ESXi Console, making sure that the console is not busy scheduling CPU time for the virtual machines. The console always uses affinity 0 to operate. When applying this method, count the number of CPUs to know how many virtual machines to configure. For example, if the server has eight cores, the maximum configuration is four virtual machines. Every virtual machine requires two dedicated CPUs. To apply the settings directly: 1. Right-click on a virtual machine in the VI Client. 2. Select Edit Settings. 3. Select the Resources tab. 4. Highlight CPU and set the Reservation field to 1200 megahertz (MHz). Verint Video Intelligence Solutions 10 of 11
5. Highlight Advanced CPU. Then under Scheduling Affinity, select Run on Processor(s) and select all the affinity fields except 0. Verifying Total CPU Use It is crucial to ensure that the virtual machines are not overloaded with CPU tasks. To verify the total CPU use: 1. Highlight the ESXi Host in the VI Client. 2. Select the Peformance tab. A graph showing such data as the Real-Time Activity for CPU, Memory, Disk is displayed. 3. Verify that the Average Percentage on the host and all cores does not exceed 55% of the total CPU. Verint Video Intelligence Solutions 11 of 11