REFERENCE ARCHITECTURE PernixData FVP Software and Splunk Enterprise 1
Table of Contents Executive Summary.... 3 Solution Overview.... 4 Hardware Components.... 5 Server and Network... 5 Storage.... 5 Test Results.... 5 SplunkIT.... 5 Splunk App for VMware.... 5 System Details and Configurations.... 6 FVP Software.... 6 Splunk Enterprise.... 7 vsphere Configuration.... 11 Conclusion....12 Appendix A: SplunkIt Test Methodology...13 Appendix B: Splunk App for VMware Reporting....13 Appendix C: Bill of Materials....14 References....14 2
Executive Summary Virtualization and cloud infrastructures offer tremendous value, but they can also introduce additional complexity and create an environment that is difficult to manage and monitor. When problems arise, finding the root cause or gaining visibility across the entire infrastructure can be a challenge. Traditional tools weren t designed for the complexity of today s environments where targeted, detailed analysis of IT and security data is required. Splunk Enterprise is an industry-leading platform for collecting and analyzing this type of data. In this reference architecture, PernixData FVP software is utilized in conjunction with Splunk Enterprise to deliver optimal Splunk indexing and search performance in a typical VMware vsphere environment. By moving storage intelligence into the server tier, FVP eliminates the storage bottlenecks that affect performance when multiple I/O intensive workloads are deployed in virtualized environments. This paper includes the results of reported storage performance as collected by the Splunk App for VMware and the PernixData FVP user interface during a series of Splunk Enterprise indexing and search performance tests. These performance tests were performed using the SplunkIt performance benchmark. For some of the tests, FVP software was deployed with the VMware vsphere stack running the Splunk virtual machines, with no changes to the Splunk Enterprise indexer, search head, or forwarder configurations. In this design, FVP software uses server-deployed solid-state disk (SSD) to increase index and search Events Per Second (EPS) performance, and eliminate the negative effects that can manifest when deploying virtual machines with multiple, varying, storage I/O patterns. No specialized storage array configurations were made in effort to represent an existing, common, virtual environment. As previously published, test results revealed that Splunk Enterprise had a light storage I/O footprint, but was very write heavy during data ingest. Additional environmental inspection revealed that the combination of write heavy indexing, read heavy search, and the varying I/O of other virtual machines resulted in increased storage latency across the entire environment. Tests with FVP software installed yielded a modest increase in index and search performance, but more importantly, eliminated noisy neighbor influences and reduced storage latency of all VMs by 600%. Results show that FVP software is well suited for Splunk Enterprise deployments as it automatically ensures newly ingested and frequently accessed Splunk data remains available in high-performance server-side flash. As data gets cold, or less frequently accessed, it s automatically evicted from flash. This makes it possible to independently scale Splunk Enterprise index and search performance based on ingest rates, while sizing storage capacity based on data retention requirements. 14 Virtual Machine Disk Latency - ms (Shorter is better) 12 10 8 6x Decrease in Latency 6 4 2 0 WITHOUT PERNIXDATA FVP WITH PERNIXDATA FVP Figure 1. Average virtual disk latency comparison 3
Solution Overview The Splunk Enterprise solution with FVP software combines the leading platform for real-time operational intelligence with the industry s premier platform for storage intelligence. By eliminating storage bottlenecks and minimizing noisy neighbor effects, the FVP-enabled solution allows for a high performance Splunk Enterprise deployment on a single shared VMware vsphere hosting platform. In addition, as requirements change, Splunk performance and capacity can be discretely scaled using a choice of many server and storage systems. Software Components PernixData FVP Software PernixData FVP software puts storage intelligence into server flash and RAM to accelerate the performance of virtualized applications. By putting storage intelligence into high speed server media, PernixData FVP software delivers the following benefits: Fast Splunk performance. By serving reads and writes locally on the host, FVP eliminates storage I/O bottlenecks, resulting in a better Splunk Enterprise and neighboring application user experience. A consistent experience, even with Splunk I/O loads. FVP software eliminates drops in performance for neighboring VMs due to Splunk Enterprise activities that stress the storage array, like indexing and searching. Newly ingested Splunk data is written and indexed from server-side flash and FVP destages this data to the storage array in a consistent, predictable stream so the storage array isn t overloaded. Extend the life of your storage device. By offloading reads and writes from the SAN/NAS to the FVP data acceleration tier, a significant amount of IOPS and throughput no longer go to the storage array. This load reduction allows the SAN/NAS to be used for what it does best (capacity and data services), extending its usable life. Deploy more VMs. Expanding the number of VMs within your current storage infrastructure is a great way to control costs. With FVP software, storage performance is based on the flash/ram on the server tier, not the SAN/NAS. This means you can deploy more VMs, not only increasing the VM density but also getting more out of your current storage array. Splunk Enterprise Software Splunk Enterprise is the easy, fast and secure way to search, analyze and visualize the massive streams of machine data generated by your IT systems and technology infrastructure physical, virtual and in the cloud. Use Splunk Enterprise and your machine data to deliver new levels of visibility, insight and intelligence for IT and the business. Splunk Enterprise can read data from virtually any source, such as network traffic or wire data, web servers, custom applications, application servers, hypervisors, GPS systems, stock market feeds, social media, sensors and preexisting structured databases. It gives you a real-time understanding of what s happening and deep analysis of what s happened across your IT systems and technology infrastructure, so you can make informed decisions. VMware vsphere vsphere is the industry-leading virtualization platform for building cloud infrastructures. It enables users to run business-critical applications with confidence and respond quickly to business needs. vsphere accelerates the shift to cloud computing for existing data centers and underpins compatible public cloud offerings, forming the foundation for the industry s best hybrid cloud model. 4
Hardware Components Server and Network Cisco Unified Computing System is the first converged data center platform that combines industry-standard, x86-architecture servers with networking and storage access into a single converged system. The system is entirely programmable using unified, model-based management to simplify and speed deployment of enterprise-class applications and services running in bare-metal, virtualized, and cloud computing environments. The computing platform with Cisco UCS used for these tests includes: Cisco UCS C220 M3 Servers Cisco UCS 6200 Series Fabric Interconnects Cisco UCS 2000 Series Fabric Extenders Storage EMC VNXe Series is an affordable unified storage platform with solution-focused software that s easy to manage, provision, and protect. The VNXe series allows you to easily deploy private clouds with deep VMware integration. It s built for trademark ease-of-use across the entire storage lifecycle - from setup to management to support. It not only provides a single platform for file and block data services with Ethernet and Fibre Channel connectivity, but data efficiency services reduce your capacity requirements up to 50 percent. The storage configuration used for these tests includes: EMC VNXe3200 Storage Array Base Capacity Bundle (36 TB raw capacity: 9 x 4 TB 7.2K RPM NL-SAS) Test Results Test results are summarized in the following section. For further details, see Appendices A and B. SplunkIt Previously published Splunk Enterprise testing has revealed that index and search performance is typically constrained by available CPU resources, so for these tests SplunkIt was used primarily as a workload generator rather than a benchmark application. Two identical standard SplunkIt benchmark tests were run to simulate a typical Splunk Enterprise workload and compare performance with and without FVP software. Test 1 was run to measure performance without FVP software. Test 2 ran with FVP software enabled using SSD as an acceleration resource. As expected, both tests were ultimately constrained by available CPU resources, but the reduction in storage latency in Test 2 (with FVP) did result in greater index and search EPS performance. Splunk App for VMware Virtual disk latency for all VMs in the environment was collected during test runs to identify the performance impact of the different configurations. The Splunk App for VMware was used to produce detailed reports based on 20-second granular metrics collected from vsphere. These reports revealed that average max total disk latency was significantly lowered in Test 2. Further inspection of VM disk latency from within the FVP user interface revealed a decrease in average latency from 12.6 ms in Test 1 to 2.1 ms in Test 2. This reduction in latency is consistent with perceivably lower application response times and increased performance of all applications in the environment. These results show that FVP software effectively eliminates the noisy neighbor effect that virtual machines with differing workload types can have on one another in virtualized environments. 5
Figure 2. Virtual disk latency detail FVP UI - Test 1 on left side of chart; Test 2 (with FVP) on right side of chart System Details and Configurations FVP Software In this configuration FVP software virtualizes server-side flash across two VMware ESXi hypervisor nodes in a compute cluster and hooks these high-speed server-side resources into existing VM I/O paths to transparently reduce the IOPS burden on an existing storage system. The FVP software installation integrates FVP-enabled VMware ESXi hosts and the PernixData FVP Management Server within the VMware vsphere platform. The installation includes three components: The PernixData FVP Management Server software, installed on a Windows server. The Management Server coordinates operations between FVP and the ESXi host computers. The PernixData FVP UI Plugin, installed on servers where the vsphere Client is installed or accessible from within the VMware Web Client. The plug-in is the primary tool used to manage FVP. The PernixData FVP Host Extension software on ESXi hosts. For a virtual machine to take advantage of FVP s acceleration capabilities that VM must be running on a host computer where the FVP Host Extension module has been installed. Using the UI Plugin, a single FVP Cluster was created and all virtual machines accelerated using datastore policies. Using this configuration method reduces configuration complexity and ensures any additional deployed virtual machines are automatically accelerated. All virtual machines were accelerated in Write Back (+1 peer for protection) mode using 800 GB of clustered SSD based flash. The following FVP configuration was used in this deployment: One Flash FVP Cluster All Virtual Machines 400 GB SSD flash per host Write Back + 1 Peer Datastore to FVP Cluster Assignments The datastores were added to FVP Clusters as summarized in Table 1: DATASTORE NAME FVP CLUSTER NAME RESOURCE TYPE RESOURCE SIZE Datastore_01 Infrastructure SSD SSD (NAND Flash) 800 GB Datastore_02 Infrastructure SSD SSD (NAND Flash) 800 GB Table 1. Datastore to FVP Cluster Assignments 6
Splunk Enterprise The core components of a Splunk Enterprise deployment include Splunk Indexers, Search Heads, and Forwarders. Splunk indexers are well suited for FVP software acceleration. In addition to rapidly writing data to disk, indexers do much of the work involved in performing searches. They read the data on disk, decompress the data, extract knowledge, and report results. Therefore, flash resource size requirements can most accurately be gauged based on indexer deployment. As the scale of data ingest increases, indexers and additional flash can be deployed to reduce contention for resources during searches, and accelerate search performance. Search heads search for information across indexers and are usually both CPU and memory intensive rather than storage I/Ointensive. Although they see only modest direct performance increases from FVP software, similar to all other VMs in the environment, they still benefit by being shielded from other noisy neighbors (like indexers). Forwarders collect and forward data to indexers. Forwarders are usually not resource intensive and were not included in this design. Ultimately, the system resources needed to enable search and index performance depend on both the volume of data being indexed and the search load. Splunk Enterprise is designed to scale horizontally and FVP software ensures that storage performance scales in unison as additional search heads or indexers are added to the architecture. This Splunk Enterprise installation included the following systems: One indexer and search head (SplunkIt) One search client (SplunkIt) One indexer and search head (Splunk App for VMware) One data collection node (Splunk App for VMware) Note: The use of the SplunkIt performance benchmark imposed specific requirements. 7
Splunk VM Configuration Virtual hardware of the Splunk Enterprise indexer and search head (SplunkIt) virtual machine was configured and optimized as defined in Table 2. ATTRIBUTE Server OS SPECIFICATION CentOS 6.4, 64-bit VMware virtual hardware Version 10 VMware Tools version 9.4.10 (up to date) Virtual CPU 8 Virtual memory 8 GB vnics 1 Virtual network adapter 1 Virtual SCSI controller 0 Virtual Disk OS - VMDK Virtual Disk Data - VMDK Virtual CD/DVD Drive 1 VMXNet3 Adapter Paravirtual (PVSCSI) 16 GB 100 GB Removed Table 2. Splunk Enterprise indexer and search head (SplunkIt) VM Configuration Virtual hardware of the Splunk Enterprise search client (SplunkIt) virtual machine was configured and optimized as defined in Table 3. ATTRIBUTE Server OS SPECIFICATION CentOS 6.4, 64-bit VMware virtual hardware Version 10 VMware Tools version 9.4.10 (up to date) Virtual CPU 8 Virtual memory 8 GB vnics 1 Virtual network adapter 1 Virtual SCSI controller 0 Virtual Disk OS - VMDK Virtual CD/DVD Drive 1 VMXNet3 Adapter Paravirtual (PVSCSI) 16 GB Removed Table 3. Splunk Enterprise search client (SplunkIt) VM Configuration 8
Virtual hardware of the Splunk Enterprise indexer and search head (Splunk App for VMware) virtual machine was configured and optimized as defined in Table 4. ATTRIBUTE Server OS SPECIFICATION CentOS 6.4, 64-bit VMware virtual hardware Version 10 VMware Tools version 9.4.10 (up to date) Virtual CPU 8 Virtual memory 8 GB vnics 1 Virtual network adapter 1 Virtual SCSI controller 0 Virtual Disk OS - VMDK Virtual Disk Data - VMDK Virtual CD/DVD Drive 1 VMXNet3 Adapter Paravirtual (PVSCSI) 16 GB 100 GB Removed Table 4. Splunk Enterprise indexer and search head (Splunk App for VMware) VM Configuration Virtual hardware of the Splunk Enterprise data collection node (Splunk App for VMware) virtual machine was configured and optimized as defined in Table 5. ATTRIBUTE Server OS SPECIFICATION CentOS 6.4, 64-bit VMware virtual hardware Version 7 VMware Tools version 9.4.10 (up to date) Virtual CPU 4 Virtual memory 6 GB vnics 1 Virtual network adapter 1 Virtual SCSI controller 0 Virtual Disk OS - VMDK Virtual CD/DVD Drive 1 VMXNet3 Adapter LSI Logic Parallel 18 GB Removed Table 5. Splunk Enterprise data collection node (Splunk App for VMware) VM Configuration 9
Server and Network Configuration Tests were performed on Cisco UCS C220 M3 rack-mount servers designed for performance and density over a wide range of business workloads from web serving to distributed database. One vsphere cluster containing both host servers was deployed and contained all Splunk Enterprise, vsphere management, and typical virtual storage load generation virtual machines. To eliminate possible CPU contention for accurate test performance comparison, vsphere DRS affinity rules were used to separate the SplunkIt benchmark VMs from all other VMs in the cluster. Each Cisco UCS C220 M3 blade was configured as follows: Two Intel Xeon E5-2640 2.0 GHz processors (16 cores total) 128 GB of RAM One VIC 1225 Dual Port 10Gb SFP+ CAN One UCSC RAID SAS 2008M-8i Mezzanine Card One 400 GB 2.5 inch Enterprise Performance SAS SSD The Cisco UCS 6248UP Fabric Interconnects used in this architecture provide both network connectivity and management capabilities for the system. They offer line-rate, low-latency, lossless 10 Gigabit Ethernet and Fibre Channel over Ethernet (FCoE) functions, and were built to consolidate LAN and SAN traffic onto a single unified fabric. Each 6248UP was additionally connected to a single Cisco Nexus 2232TM Fabric Extender to facilitate direct connection of the VNXe32000 s 10GBase-T interfaces to the unified fabric. Storage Configuration Capacity The EMC VNXe3200 unified hybrid storage system used in this architecture brings the power of EMC s VNX to the IT generalist. It retains the affordability, simplicity, and efficiency of previous generation VNXe systems and adds support for MCx multicore optimization, FAST Cache SSD caching, FAST VP auto-tiering, and Fibre Channel host connectivity. These enterprise-class features were previously reserved for higher-end storage systems. A single, redundant, VNXe3200 array was used for testing. The system was connected to the storage fabric by four 10 GbE (one 2-port LACP bond per SP) connections utilizing iscsi. The following components made up the VNXe3200 deployment: One RAID 6 (6+2) Storage Pool (4 TB, NL-SAS disks) Two 1 TB LUNs Both hosts were zoned and masked to allow access to two 1 TB VMFS (block) iscsi datastores. Performance PernixData FVP Software was used in this architecture to serve reads and writes locally on host servers and eliminate storage I/O bottlenecks. In this design, FVP software uses 400 GB 2.5 inch Enterprise Performance SAS SSDs to accelerate all virtual machines. The following FVP configuration was used in this deployment: One flash based FVP Cluster All Virtual Machines 400 GB SSD flash per host Write Back + 1 Peer 10
SSD sizing for this reference architecture was based on the data ingest size in phase one of the SplunkIt performance benchmark and may be appropriate for many standard Splunk Enterprise deployments. For these tests, 400 GB of flash per host was more than adequate to hold not only hot index and search data for the Splunk Enterprise virtual machine, but frequently accessed data for other VMs in the environment as well. For environments with higher ingest rates or deeper search requirements, flash size can be increased accordingly based on Splunk Enterprise VM hit rate as reported by FVP. vsphere Configuration One virtual data center was deployed for all virtual machines. This is the standard, recommended configuration for most production deployments. Infrastructure and Management Configuration A single VMware vcenter Server instance was deployed for all desktops and server virtual machines. For simplicity, all vcenter roles (inventory, SSO, vcenter) were consolidated onto a single VM. No resource contention was observed on any parts of the management infrastructure during tests. All server resources were sized according to the current best practices from VMware. They are listed in the following table: SERVER ROLE VCPU RAM (GB) STORAGE (GB) OS Domain Controller 2 4 40 Windows Server 2012 64-bit R2 SQL Server 2 8 40 Windows Server 2012 64-bit R2 vcenter Server 2 8 40 Windows Server 2012 64-bit R2 Table 6. vsphere Server Resource Sizing Typical Virtual Storage Load Generation Configuration An additional virtual machine was deployed in the cluster to simulate the storage load induced by a typical virtual server deployment. A VMware I/O Analyzer Appliance was used to run a mix of 8K and 16K, 60% random, 75/25 read/write storage workloads for the duration of all tests. The appliance specifications are listed in the following table: SERVER ROLE VCPU RAM (GB) STORAGE (GB) OS Storage Load Generation 4 2 32 SLES 64-bit Table 7. Server Storage Load Generation Resource Sizing 11
Conclusion Tests clearly show that PernixData FVP software is a perfect complement for high performance, low latency Splunk searches in VMware vsphere environments. An FVP-enabled architecture automatically ensures newly ingested and frequently accessed Splunk data remains available in high-performance server-side flash for the best index and search performance. Contrary to traditional storage capacity/performance coupled designs, this also makes it possible to independently scale Splunk Enterprise index and search performance based on ingest rates, while independently sizing storage capacity based on data retention requirements. When virtual environments are properly designed with FVP software, storage performance is optimized and noisy neighbor affects are eliminated, allowing for maximum performance, flexibility, and cost savings. When designing this PernixData FVP Software and Splunk Enterprise reference architecture, specific hardware components were selected to accurately represent an existing, well-designed, virtual environment. However, it s important to note that although these components were used to test and validate Splunk Enterprise performance with FVP, they are not specifically required. Decoupling storage performance from capacity using FVP software allows for the use of many different hardware platforms designed to run VMware vsphere. The results of this document should be used alongside Splunk Enterprise best practice documents to help choose hardware based on actual environmental requirements. 12
Appendix A: SplunkIt Test Methodology SplunkIt is a performance benchmark kit designed to provide a simplified set of performance measurements for Splunk. SplunkIt provides administrators a straightforward benchmark tool to test infrastructure configurations. Because the benchmark performs all phases of a typical Splunk Enterprise deployment, it serves as an excellent application to simulate typical Splunk Enterprise workloads. The kit creates a small dataset comprised of syslog events (50 GB) that can be leveraged for measuring the following performance characteristics: Indexing throughput Search response time The benchmark consists of three main phases: 1. Generate Data - Generate 50 GB of syslog data 2. Index Test - index 50 GB of syslog data 3. Search Test - Search using Splunk Web For the tests outlined in this design, a Splunk Enterprise VM was created and a snapshot taken prior to starting phase one. At the completion of all phases, the VM was rolled back and the tests performed again with FVP software enabled. Appendix B: Splunk App for VMware Reporting The Splunk App for VMware provides real-time operational performance insights. It provides immediate visibility into the health of the VMware environment and instantly identifies VM, host and datastore problem areas. For the tests outlined in this design, the Splunk App for VMware was used to generate reports based on the 20-second granular metrics collected from vsphere below: latest_disk_maxtotallatency_millisecond These reports help identify over-utilized storage resources and gauge virtual machine performance. Data was collected for all virtual machines in the environment and results compared for times in which the SplunkIt benchmarks were executed with and without FVP software enabled. 13
Appendix C: Bill of Materials The test configuration bill of materials is summarized in the following table: AREA COMPONENT QUANTITY Host Hardware Cisco UCS C220 M3 (2 x Intel E5-2640, 128 GB RAM) 2 Storage Hardware EMC VNXe3200 Array Base Capacity Bundle (9 x 4 TB, 7.2k RPM NL-SAS Disks) 1 Network Hardware Cisco UCS 6248UP Fabric Interconnect 2 Cisco Nexus 2232TM Fabric Extender 2 PernixData FVP 2.0.1.0 2 VMware ESXi 5.5 Update 2 2 Software VMware vcenter 5.5 Update 2c 1 Splunk Enterprise 6.1 50 GB Microsoft Windows 2012 R2 3 Microsoft SQL Server 2012 SP1 1 Table 8. Bill of Materials References PernixData Product Page: http://www.pernixdata.com/products Splunk Enterprise Product Page: https://www.splunk.com/en_us/products/splunk-enterprise.html VMware Product Page: http://www.vmware.com/products SplunkIt Product Page: https://apps.splunk.com/app/749/ Splunk App for VMware Product Page: https://apps.splunk.com/app/725/ www.pernixdata.com 1-855-PERNIX-D @PernixData Copyright 2015 PernixData, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. PernixData is a registered trademark and FVP, FVP Cluster and Flash Cluster are trademarks of PernixData, Inc. in the United States and/or other jurisdictions. All other brands, products, marks and names mentioned herein may be trademarks or service marks of, and are used to identify, products or services of their respective owners. 14