Top 10 Reasons Why Customers Choose Load DynamiX over Freeware Testing Tools

Transcription

1 Top 10 Reasons Why Customers Choose Load DynamiX over Freeware Testing Tools Get more realistic storage performance testing of application workloads for better decision-making January Load DynamiX. All rights reserved.

2 Table of Contents I. Introduction... 3 II. Ease of Use / Greater Productivity... 4 III. Comprehensive Reporting... 7 IV. Performance... 9 V. Professional Support VI. Best-in-Class Test Management VII. Best-in-Class Vendor Collaboration VIII. Best-in-Class Realism IX. Consistency and Repeatability of Results X. Broad Coverage for File, Block, and Object Storage XI. Lower Total Costs XII. Summary Load DynamiX. All rights reserved. LoadDynamiX.com 2

3 I. Introduction Load DynamiX brings advanced workload modeling and storage infrastructure performance validation to both technology vendors and IT organizations. Our solutions provide critical insight to help our customers optimize the performance, availability, and cost-effectiveness of their storage systems. Load DynamiX solutions enable the accurate emulation of production application workloads at extreme scale to optimize storage system deployment decisions and find storage system limits before being deployed in production. Nearly all of our customers used freeware tools like Iometer, Vdbench, and fio at one time. They moved from these tools to Load DynamiX for some or all of the 10 reasons discussed below. This paper explores each area and explains how we achieve each of these differentiators. 1. Ease of use & deployment: results in up to 10X more tests per month for higher storage engineering productivity 2. Comprehensive reporting tools: greatly reduces the time It takes to generate, compile and share reports 3. Professional support: you ve got an active partner in solving problems and evolving the product 4. Performance: purpose-built load generation appliances optimized to generate and drive extreme storage workloads across a wide variety of storage protocols and interfaces. 5. Best in Class Test Management: single pane of glass and global sharing of test functions to simplify the storage testing process. 6. Best in Class Vendor Collaboration: enabled by using identical load generators and workload models combined with an Integrated database across all data centers and testing labs 7. Best in Class Realism in workload profiling: results in better purchase and deployment decisions that minimize cost and assure performance 8. Repeatability and Consistency of results: so you have 100% confidence in all comparisons 9. Broadest coverage: File, Block and Object - learn and use one testing platform instead of multiple different tools 10. Lower total testing costs: substantial savings on people and capital expenses 2015 Load DynamiX. All rights reserved. LoadDynamiX.com 3

4 II. Ease of Use / Greater Productivity The Load DynamiX Enterprise software platform is an intuitive web UI built for all user levels. It controls multiple appliances that can be leveraged by defined user groups. Simple and easy to use, the latest UI product comes preconfigured with a test platform, protocol test suites, and test content. Beyond just providing a GUI, the design offers many benefits over the simple freeware tools on the market, including: Simple Collaboration Share Load DynamiX resources such as appliances, ports, tests, and test results with specified groups or individuals based on their role within the organization. Access to specific resources and tests can be dynamically determined to support higher levels of security and efficiency. Such functionality allows team members to work more effectively with each other and increase the ROI in Load DynamiX appliances. One of the biggest benefits is the ability to automate, with error detection and flagging. And, overall, for a single test case, we find at least 10x improvement now minutes compared a half day or more. John Bothe Chief Architect Exinda Sample Workload Models The following sample workload models are shipped with the current version of Load DynamiX Enterprise: iscsi, FC, NFS and SMB versions. These workload models allow users to define and control key IO access patterns with a simple Web interface, and deploy them to a test bed. Application Workloads Library Load DynamiX Enterprise provides a set of customizable workload models that are characterized and pre-configured for specific applications, such as OLTP, VDI and various other workloads. Such valuable assets allow users to generate realistic workloads without the need to do extensive research and manual data collection Load DynamiX. All rights reserved. LoadDynamiX.com 4

5 Further, Load DynamiX offers a single solution for File, Block and Object instead of several tools and separate hardware, and offers a clear advantage over the learning curve that goes with using multiple tools. Simple Test Execution of Multiple Tests Run and configure tests with a simple push button interface. Load DynamiX Enterprise allows engineers to configure and execute tests without requiring protocol expertise or extensive scripting. For example, Workload Iterator, a Load DynamiX Enterprise utility, allows users to easily automate iterations over hundreds of parameter permutations that define many different characteristics of the workloads. These characteristics include block sizes, read vs. write percentages, sequential vs. random percentages, number of files and folders, LUN hot spots, load profiles, deduplication and compression ratios, FC queue depth, and more. You can run hundreds or even thousands of tests from a single command, and differentiates Load DynamiX from freeware tools. Having Load DynamiX has improved my job tenfold. I m now free to do things more important and more technically challenging than running around trying to start a bunch of clients simultaneously. Nigel Stolting QA Manager Hitachi No other storage testing product can so quickly identify strengths and weaknesses, and decrease overall testing time. There s no need to write, spawn, and track results from hundreds of separate test/threads. Today, freeware tools require extensive scripting to come close to this level of automation. The high end personnel resources that are dedicated to scripting can be freed up to do the kinds of analysis that actually provides real business benefits. With the greater productivity of Load DynamiX, customers are able to implement the backlog of tests that tends to grow ever larger over time Load DynamiX. All rights reserved. LoadDynamiX.com 5

6 In the figure below, from a single input screen, we are running 60 tests with different parameters including various block sizes and concurrent sessions, to determine the optimal queue depth settings on a fibre channel SAN, for the application workload on the target array. No scripting required, and it could just as easily be thousands of tests. But the real freeware alternative is not scripting. Frankly, it takes so long to set up so many iterations without this sort of iteration utility, that these kinds of performance profiling tests simply are not done. Figure 1: Simple setup for running 60 tests, with 11 input parameters to find optimal queue depth settings Load DynamiX. All rights reserved. LoadDynamiX.com 6

7 III. Comprehensive Reporting Load DynamiX reporting removes the need to script, compile & analyze results from dozens of clients and tools. There s no need to find the ASCII or html output files from each load generator / server, and consolidate them into a format that you can use. Load DynamiX Enterprise allows you to easily share the results of your tests by creating a reusable report template as the basis for generating reports. To create a template, you select that option from the dashboard, name and describe it (see Figure 2 below) then select the statistics of interest from the dropdos (see Figure 3 below), select the chart type (see Figure 4 below), and you re done. To use that template, you simply open it, point it to the workload(s) you re interested in, name it, and run it. All reports are sharable as PDFs, CSV files, or even as URLs. Figure 2: Naming a Report Template and selecting the Report Type Load DynamiX. All rights reserved. LoadDynamiX.com 7

8 Figure 3: Selecting the data of interest from the Reporting Template. Figure 4: Selecting the Chart Type from the Reporting Template Load DynamiX. All rights reserved. LoadDynamiX.com 8

9 IV. Performance The Load DynamiX load generation appliances are purpose built devices and software that have been optimized to generate and drive extreme storage workloads across a wide variety of storage protocols and interfaces. The Load DynamiX solution combines a realtime operating system with a polled testing model, and no interrupts (that you find on general purpose servers used as load generation platforms), to control the operation of a test at nanosecond precision and ensure highly repeatable results. You can run many tests at the same time, without interfering with other tests. In a general-purpose load generation environment, you must be careful to run tests serially. With the scale and performance of Load DynamiX platforms, your utilization of all resources and testing productivity will be increased many times over. We can now run tests that were very hard to do before. We couldn't test 10,000 SMB connections quickly using our in-house tools. Now we can." Ken Sarno QA Manager Riverbed Realistic test results must be derived from a load that simulates the production workload. It s not uncommon for freeware tests to be under-provisioned, so test results are often extrapolated from insufficient test results. A single Load DynamiX 2U appliance can generate the equivalent of over 10 general purpose servers in a lab environment. As an example, the IOPS figures below were generated to offer guidance on expected IOPS performance for each selected Load DynamiX appliance, at various block sizes, in a 16G fibre channel environment. Similar guidelines are available for all supported protocols. There is no analogous guidance in the freeware / general purpose server rack environment Load DynamiX. All rights reserved. LoadDynamiX.com 9

10 Figure 5: Fibre channel appliance performance statistics Load DynamiX. All rights reserved. LoadDynamiX.com 10

11 V. Professional Support Load DynamiX provides outstanding customer support, professional services and education services with a team dedicated to performance testing and validation. You don t have to rely on a public forum and unpaid volunteers for answers to your questions. Professional Services provides everything from phone assistance to workload modeling and training to turn-key storage testing and evaluation solutions. They have the expertise to make networked storage testing and performance validation simple and cost effective. And thanks to their dedication, we have nearly a 100% customer support renewal rate every year! Unlike freeware tools that have updates every year or so, Load DynamiX delivers frequent major product updates, ensuring that our customers keep current with new technologies, protocols, etc. Storage networking technology has advanced significantly in the last five years after a more stable period with less innovation. The limited advances in disk drive technology reduced differentiation among vendors. The advent of Solid State storage technology (flash) combined with higher processor capabilities have enabled not only much higher transaction rates, but have also data reduction technologies such as compression and deduplication that can dramatically reduce storage requirements. Additionally, virtualization dramatically alters and increases the variability of storage traffic patterns. All of these can significantly impact performance. I can t say enough about Load DynamiX s support and engineering teams. They have an extremely intelligent and experienced staff. Brian Walker Principal Architect, Cloud Solutions GE As a result, choosing the best storage solution for applications now requires much more than comparison by reading articles or studying collateral. Testing is a requirement for choosing storage technologies, optimizing storage configurations, breaking in new technology / solutions and managing change Load DynamiX. All rights reserved. LoadDynamiX.com 11

12 VI. Best-in-Class Test Management Load DynamiX Enterprise is an easy-to-use software platform with a single point of management for characterizing and creating workload models, configuring and administering tests, and analyzing results. From this single point, which is accessible to your entire team, you can: Manage many concurrent applications and workload emulations Provide a centralized reporting repository Manage your historical database of tests, workloads, templates and reports. In the extract of the Enterprise Dashboard below (Figure 6), anyone with authority can view this information from anywhere in the world, for test labs running any number of Load DynamiX appliances. In this screen shot, under test results, you can see that there is one test running, and 3,734 completed tests, all available for historical viewing. As an aside, this shows one test but you could easily be running many tests at the same time and have a status of each one. A typical use case is one where someone would find the workloads associated with completed tests, and run those workloads on a new system or configuration. In that way, an apples-to-apples comparison is possible for optimizing configuration changes or testing new firmware releases, as two examples. Figure 6: Load DynamiX Enterprise test management console Load DynamiX. All rights reserved. LoadDynamiX.com 12

13 VII. Best-in-Class Vendor Collaboration An important benefit to running tests with Load DynamiX is that there s an excellent chance that your vendor also runs Load DynamiX, and can repeat your tests in their labs. All major and nearly all smaller storage vendors have standardized on Load DynamiX in their performance engineering and QA labs. We can point to examples of where a performance issue was uncovered at a customer site and their Load DynamiX workload was sent to their vendor, who ran the exact workload in the vendor s lab. In these cases, the vendor was able to quickly respond with a fix or an explanation for what was causing the problem. In some cases, the problem was uncovered by running a Proof of Concept (POC) with Load DynamiX, and by sharing he results with the vendor, the vendor was able to repeat the exact tests in the lab and provide a fix or workaround during the customer s POC evaluation. The result is a better, more informed buying decision. We never could have reproduced the customer's workload, at such scale, without the help of Load DynamiX. Brian Hudson Systems Engineer EMC VIII. Best-in-Class Realism One of the values of lab testing relies on how closely you can emulate actual production workloads. More realistic emulation results in better purchase and deployment decisions that minimize cost and assure performance. A richer set of test parameters more closely simulates your actual production environment. Factors affecting realism include the following: Metadata Many freeware tools do not include Metadata, we suspect, because it s not real data. A prime example of Metadata are file system calls in SMB and NFS workloads. But in real-world applications, Metadata can constitute well over 50% of all I/Os, and some customers report the number to be over 90%. Without it, results are highly inaccurate and will lead to incorrect conclusions. With Load DynamiX Enterprise, it s a simple matter to configure data along two axes: Data and Metadata. The Data configuration controls the overall percentage of Reads and Writes that the workloads execute Load DynamiX. All rights reserved. LoadDynamiX.com 13

14 The Metadata configuration controls the overall percentage of non-data related operations that the workload executes. The sum of the Data and Metadata percentages must add up to 100%, as demonstrated in Figure 7, below. Figure 7: Specifying metadata as a workload parameter in Load DynamiX Enterprise. The figure below shows an example of the actual breakdown of command mix count. In aggregate, the metadata commands constitute a large percentage of I/Os. Figure 8: Load DynamiX Enterprise command mix count Load DynamiX. All rights reserved. LoadDynamiX.com 14

15 Spatial and Temporal Localities There are two basic types of reference locality temporal and spatial locality. Temporal locality refers to the reuse of specific data, and/or resources, within a relatively small time duration. Spatial locality refers to the use of data elements within relatively close storage locations. Many applications exhibit high degrees of these localities. For instance, search demonstrates significantly higher spatial locality, with some portions of the disk being frequently accessed and others heavily underutilized. Another instance is SSD caching, which takes advantage of frequently accessed blocks to improve performance. Temporal locality is commonly exploited by the use of lookup tables and caches. Traditional storage I/O workload generation tools lack the ability to detect and utilize the spatial and temporal locality of I/O access patterns, causing them to significantly deviate from the application's indigenous characteristics. Using the Load Dynamix Test Development Environment (TDE), the user can develop application workloads that simulate both spatial and temporal localities. Those workloads can then be executed on LDX-Enterprise. Figure 9 on the next page represents temporal behavior of the Read and Write chunk sizes created In TDE in user parameter files generated from statistical data of an actual production workload. Often, without this kind of temporality, you can lose a lot of important insight to averages. 250 Read chunk size variations Read chunk size (KB) time (min) Figure 9: Minimum averaged Read chunk size variations in Load DynamiX Enterprise Load DynamiX. All rights reserved. LoadDynamiX.com 15

16 Composite Workloads With some traditional workload benchmarking methods, workloads and storage protocols are validated one at a time, independently. The Load DynamiX Enterprise Composite Workload feature allows the user to create many different workloads, across different storage protocols, and run them all at the same time. This results in a much more realistic and holistic emulation of production environments. In figure 10 below, a realistic Oracle database application emulation is achieved by profiling each LUN and running each profile as a single composite workload. Using the Enterprise GUI, each of the two client/ports is mapped to the set of four LUNs (color coded in blue, green, orange, and purple) with common I/O profiles and run in aggregate, resulting in an accurate portrayal of the performance of that Oracle based application. Figure 10: Load DynamiX Enterprise GUI setting up a composite workload Load DynamiX. All rights reserved. LoadDynamiX.com 16

17 Precise Control Realistic modeling is largely about control of the test variables at scale. In a recent vendor comparison, a Load DynamiX customer was not only interested in how the proposed vendor storage systems performed at the current workloads, but was interested in what if scenarios. One of the reasons Load DynamiX was selected over Iometer was our ability to simulate a 2X, 3X, and 4X load by increasing the number of IOPS. This ability was absolutely crucial to the selection of the optimal vendor/system. There is no IOPS input parameter with Iometer. Many of our customer s power users take advantage of our TDE (Test Development Environment). TDE is a Load DynamiX client application for designing and executing detailed tests as well as validating test results. It s included as a component of our solutions. It s worthwhile noting that Oracle is the developer/owner of one of the well-known freeware performance tools, Vdbench; yet Oracle is one of our top customers. One of the reasons is the precise level of control that Load DynamiX offers. Aggregation of results is our enemy. We want to see those results from individual components as they all have differing effects and interactions on a storage array. With Load DynamiX we can trace, track, and compare anything all the protocol-level details we want. Steve Downer Director of Test Engineering Oracle Key features of the TDE include: Flexible Test Modeling with Test Scenarios and Control Actions Test scenarios allow protocol behaviors and sequences to be modeled on top of network profiles. Test scenarios can operate independently on the same physical interface or they can be synchronized using control events. Each test scenario can be configured with low-level command sequences available through protocol suites and 2015 Load DynamiX. All rights reserved. LoadDynamiX.com 17

18 control actions. Control actions allow the test engineer to insert loops within the tests, influence test timing by inserting delays into scenario execution paths, and advancing or resetting the next element pointer for user parameter files. User Parameter Files and Functions for Realism User parameter files hold values that can be delivered to the input fields of Load DynamiX commands while those commands are executing. This capability allows a test engineer to create test templates that are provided with the information needed to operate at run time. Thus, a single template can be used to simulate variable numbers of unique users or create variable numbers of unique files or directories. Functions can also be used in place of constant values to randomize or control the values used within commands. As previously noted, using the Load Dynamix Test Development Environment (TDE), the user can develop application workloads via user parameter files and functions that simulate both spatial and temporal localities. Those workloads can then be executed on LDX-Enterprise. Modeling Data Reduction A key feature of today's all-flash storage array implementations is the use of data deduplication and compression. Deduplication and compression are technologies that dramatically improve IT economics by minimizing storage requirements, backup windows, and network bandwidth consumption. These technologies are particularly attractive in the solid state storage space to enhance solid state storage financial viability and extend their use to a greater number of application workloads Today s storage systems (all-flash arrays or hybrid storage), are more sensitive to the workloads and apps storing data on them than the disk-based systems of old. Storage architects need to test their systems with workloads that closely model their real world applications. With Load DynamiX, architects can easily identify performance limits for their unique workloads and determine the strengths and weaknesses of any networked storage array. Howard Marks Chief Scientist Deep Storage, LLC 2015 Load DynamiX. All rights reserved. LoadDynamiX.com 18

19 beyond the Tier-1 applications where they are most commonly initially deployed. But you can t treat dedupe/compression like a checkbox. Implementations vary as well as the effect on different workloads. Load Dynamix is uniquely positioned to properly test and validate modern storage technologies like deduplication and compression with high accuracy, tremendous load and extremely realistic workload patterns. We use a patent pending method of generating and verifying streams that represent repeatable random and repeatable compressible content. These algorithms can generate exabytes of unique data, enabling testers to specify the wide range of patterns and repetition required to properly exercise data reduction technologies, thereby helping to test arrays from any vendor that implements the features. We generate a combination of completely unique patterns and repeating patterns. Any repeating pattern should reduce down to one deduplicated entry. We do the same with compressible data, either unique or repeating. And we combine those data types to generate a data set and the data streams required to properly test an array. Load Dynamix uses multiple Write statements, for the selected protocol, with the capability of selecting the appropriate data type and the number of times repeated to lay down a dataset via a data stream that emulates real world traffic. The storage array then executes the deduplication and compression based on those patterns, stores as metadata, and we measure the results on metrics like IOPS. Figure 11: Load DynamiX data showing IOPS v Read/Write ratios. In figure 11, above, we demonstrate that different arrays will perform differently using different amounts of repeating data. In this example, vendor B exhibits higher IOPS when data is 20% or 50% reducible, 2015 Load DynamiX. All rights reserved. LoadDynamiX.com 19

20 with any read/write ratios, but especially with higher reads. Vendor A shines when data is highly reducible and read/write ratios are in the 50% range (or greater). Most freeware tools do not enable the Storage Engineer to offer this level of analysis. Load DynamiX differentiates itself by articulating a proven methodology for building a realistic performance model, which we have summarized in the following section entitled Building a performance model. The more complete methodology is available in a separate document. Building a performance model There are 4 basic areas to consider when characterizing a workload for an environment: 1. The description of the size, scope and configuration of the environment itself, 2. The access patterns for when, how frequently and in what ways the data is accessed, 3. The load patterns over time, and 4. The impact within the network subsystem itself. Though this information isn t utilized directly to model the workload, it enables an impact comparison to be made between the emulated traffic and the actual production traffic. This can serve as a measure to determine how representative the emulated traffic is to real world traffic. 1. The Environment Itself Understanding the environment itself differs for file, block, and object storage. Each of these has unique characteristics that need to be understood in order to create and map to a similar emulated environment. File (NAS) For file, the following are important environmental characteristics that should be determined: The number of clients and servers The number of clients per server The authentication medium for clients to gain permission to access the servers Shares and mount points The file size distribution Distribution of sub directories and tree depths Average number of files per directory and server 2015 Load DynamiX. All rights reserved. LoadDynamiX.com 20

21 Block (SAN) For block, the following environmental information should be determined: The number of physical initiators in the environment (HBA or NIC s) The average number of virtual initiators per physical initiator The average number of active virtual initiators per physical port Number of Logical Units (LU s) per HBA Distribution of LU sizes The initiator to target fan in ratio (the average number of initiators that access each target port) The queue depth setting in the environment for the server HBAs or iscsi LUN Object Numbers of clients and servers Average open connections Authentication medium for data access Object size range The above data can be obtained by running statistics commands that each of the storage vendors support for their storage arrays. 2. The access patterns Access patterns are key to understanding how frequently and by what means the storage is accessed. This is the area where it is most important to consider several use cases such as average, peak, special business process, and events. The proper characterization of the access patterns is different for file, block, and object environments. File (NAS) For file access it is important to consider the breakdown by percentage for common commands. Often there are far more metadata commands in an environment than those that read and write actual data. The most important commands to analyze for each protocol version are: 2015 Load DynamiX. All rights reserved. LoadDynamiX.com 21

22 NFSv3 SMB2 Read % Write % Lookup % GetAttr % ReadLink % ReadDir % Create % Remove % FSSTAT % SetAttr % ReadDirPlus % Access % Commit % Read % Write % Create % Query Dir % Query Info % Set Info % Flush % Close % Block (SAN) For block access, there are far fewer metadata commands relative to file access. This is because the metadata is primarily abstracted by the file system (or data structure for raw storage) so queries to discover file layout turn into actual reads. In order to characterize block data it is important to understand basic command mix, whether the data is accessed sequentially or randomly, the I/O sizes, any hotspots, and the data content. The following parameters can be used to describe the block access patterns: Read / Write ratio Read Random / Sequential ratio Write Random / Sequential ratio Distribution of I/O request sizes (block sizes) Distribution of I/O request locations [logical block address regions (LBA)] Data access Hot spots Read and Write latencies Data content compressibility (repeating patterns) Data content de-duplicability of the store data (duplicate files or file segments) 2015 Load DynamiX. All rights reserved. LoadDynamiX.com 22

23 Object Database transactions (Short writes and reads) Database analytics (Long reads) Personal storage and backup (Long writes) Document access (e.g. SharePoint, long reads and writes, short reads and writes) 3. The Load Patterns The load patterns help to determine how much the demand and load fluctuate over time. In order to generate a real world workload, understanding how the following characteristics vary over time is the remaining piece to the modeling puzzle. The following characteristics are common to all protocols although they may have slightly different meanings. IOPs per NIC or HBA IOPs per application type Read IOPs Write IOPs Metadata IOPs Total Bandwidth (MBs) Read Bandwidth (MBs) Write Bandwidth (MBs) Data content compressibility Number of Connections In addition, each protocol has characteristics unique to the protocol that are necessary for comprehensive characterization. File (NAS) NFS Number of Locks Metadata Bandwidth (MBs) SMB2 Number of Sessions Number of Open Files Number of Locks Number of Notifications Metadata Bandwidth (MBs) Block (SAN) Number of pending/queued exchanges Number of reservations Object Load Dynamix is currently working with SNIA and leading storage architects to build a model that accurately describes the load patterns of object-based stores Load DynamiX. All rights reserved. LoadDynamiX.com 23

24 4. Impact Characterization It is often useful to observe the impact of the load pattern affect over time on the following components within the subsystem. Controller or Server CPU utilization Storage access by Tier for Tiered storage Cache hit ratio Raid type weighed hit ratio Sequential / Random access % (may differ from the requestors perspective) Latency By comparing these values from the production environment to those in the emulated environment one can access how closely the impact to the storage subsystem matches the impact to the production environment. How to characterize workloads Generic Workloads Generic application workloads are workloads that are created by Load DynamiX based on our observation of an application s behavior in one or more environments. Generic workloads are most useful for certain impactful events such as login/logout storms, VM migrations, or other application specific activities that will be relatively common across customers and environments. These workloads can be characterized by client or user and then scaled to match the expected environment. They should also often be combined with normal workload activities for an environment. Generic workloads for specific applications are often useful for storage vendors who are looking to understand as a rule of thumb how certain applications behave. For enterprise IT organizations the application specific generic workloads are not usually close enough represent the actual workload and its best to develop a customer specific application workload. Load DynamiX offers a number of generic workloads and is continually expanding the applications and events for its generic workload packages. Impactful Events Impactful Events can often be scaled up and scaled down to match an environment and behave reasonably close to what would be expected in a production environment. When combined with 2015 Load DynamiX. All rights reserved. LoadDynamiX.com 24

25 customer specific workloads, these event workloads can help determine how an environment will react to uniquely stressful events often caused by certain applications. Application Specific Workloads The application specific workloads (we will address composite workloads later in this document) are made up of combinations of observed workloads in real world environments or as specified by the application developers. There will often be more than one workload for each application if there are common application specific stressors like year or month end close or backups. Some application environments such as web servers will show more commonality and can be used more broadly than other applications like databases, which can differ greatly from implementation to implementation. One approach is to group like individual LUN activity where there is commonality between LUNS (i.e. redo logs that behave the same way) and create a workload profile for each group. Short term trace captures Short-term trace captures can be obtained from any environment, by utilizing network protocol analyzers. There are free analyzers available that work by using NIC cards in promiscuous mode (observing all traffic on a link whether it is sent to a device or not) such as Wireshark, or by utilizing hardware protocol analyzers that offer greater flexibility to filter and parse data enabling longer periods of time to be analyzed. Captures can sometimes also be obtained by working with storage vendors to turn on their ability to capture all the traffic they receive. Short-term trace captures can be good for characterizing unique events that occur in environments but are generally too short to get an overall picture of how an application or network segment typically behaves. The data generated by these short trace captures is immense, therefore, it is very expensive when used for capturing long periods (a day or longer) of traffic. Load DynamiX has utilities to turn many of these captures directly into the necessary characterizations needed to model the workloads. Information gathered by the storage subsystem Many storage arrays continually collect much of the information necessary to fully characterize both the Access Patterns as well as the Load Patterns (except temporality). In most cases it is best to consult with your storage vendors to determine the best approach to obtain both the Access Pattern information as well as the Load Patterns Load DynamiX. All rights reserved. LoadDynamiX.com 25

26 IX. Consistency and Repeatability of Results With traditional testing tools it s well known that test results can t be reliably repeated. Many academic papers have been written on this subject and the given advice is to run the test multiple times and take the average as your result. The reason for this is not hard to understand. Freeware tools are all deployed on general purpose computers and their VMs. General purpose computers are just that - general purpose - and experience all kinds of interrupts for internal housekeeping functions simply to maintain their health. Having said that, tests run back to back should result in pretty close numbers. And pretty close may be close enough for a given scenario. Now, change your load generation rack to switch out a problematic system for example, or let vsphere move resources around, and you ll start to see more variability creep in. Lose a couple of systems and replace them 3 months later, and you probably can t even acquire the same hardware. Or the OS has been updated, or new firmware has been made available. Or if you re doing POCs at vendor sites, their racks of load gen servers are for sure different than those at other sites / other vendors. Or you run a test and want to repeat it on another rack/system in a year in another country. Net/net: there s no real repeatability or consistency. With Load DynamiX, repeatable tests and loads are executed by a purpose-built, optimized appliance and software stack. Not only is the hardware purpose-built with exacting specifications, but so is the operating system, and even the TCP/IP stack was custom built. Tests are fully automated and runs & re-runs and aggregated output can be easily and reliably compared now and in the future. This enables true apples to apples comparisons and eliminates the guesswork you re trying to avoid by doing testing in the first place. X. Broad Coverage for File, Block, and Object Storage Unlike freeware testing tools, Load DynamiX offers detailed performance emulation of a very wide band of storage protocols that provide rich, accurate emulations of workloads across File, Block and Object / Cloud storage. Setting up repeatable tests in Load DynamiX saves time. There is no need to setup multiple servers / VM s, configure operating systems or applications. Tests can easily be switched back and forth between devices under test. The consistency of the Load DynamiX solution across multiple storage networking approaches and protocols enables honest comparison of different storage networking approaches. Block, file and object solutions can all be easily compared, reducing test preparation time and enabling insight through a consistent platform with a single pane of glass that enables complex storage technology analysis without complex results configuration Load DynamiX. All rights reserved. LoadDynamiX.com 26

27 Supported protocols include: SMB, SMB2.x, SMB 3.0 dialect, MS-RPC, NFSv2, NVSv3, NFSv4, NFSv4.1, iscsi, Fibre Channel, HTTP, HTTPS, OpenStack Swift, SNIA CDMI, and Amazon S3. XI. Lower Total Costs Freeware software is free, while Load DynamiX is a commercial enterprise, focused on providing a solution that offers a greater ROI and lower TCO than freeware tools, which are combined with racks of servers and VMs. No matter how free the software is, you still need to provision, staff, and support a testing lab. One Load DynamiX 2U appliance can do what physical servers can do, which saves on setup, administration, virtualization software licenses, power, cooling, & floor space. With Load DynamiX, engineers can spend 80-90% of their time doing actual testing, as opposed to spending 80-90% of their time administering their test lab. In the freeware TCO, in addition to the $0 of freeware, you can easily spend tens or hundreds of thousands for a rack of servers, OS costs, and supporting applications, plus tens of thousands per year on environmental / space costs, plus tens of thousands for virtualization software and support, plus hundreds of thousands per year for the setup / deployment / reporting engineering efforts to run major tests. We re saving significant dollars on hardware, cutting rack space, power, and cooling costs. And capital investment is smarter. Instead of buying the bunch of NFS client test machines that were on my purchasing roadmap, I got the Load DynamiX appliance. I now need only 2U for every 8U I needed before. Bryan Schmersal Technical Staff Member Avere This is no hyperbole; traditional testing methods require a cumbersome setup of hypervisors, VMs, and applications. Every change to the test configuration, such as an increase in the number of VMs or a change of a guest OS, requires a significant amount of work. Moreover, the approach limits evaluation to the available test hardware, which may not be sufficient to run hypervisors with the hundreds of VMs that may be required to exercise the limits of the server. It s easier to justify ROI for a dedicated 2015 Load DynamiX. All rights reserved. LoadDynamiX.com 27

28 test/dev lab as Load DynamiX is a load generator in a box. The true cost of this effort shows up in two places. First, you ve got highly paid engineers spending time working on test plumbing, rather than actually running tests or better yet, valuable analysis. And second, because it s so hard, you simply don t do the number and kinds of tests you d like to do. It s not unusual for test labs to have substantial backlogs of tests, and simply not do the tests they d like. With Load DynamiX, you re saving on the servers and racks procurement and support, environmentals, virtualization licenses and support. And in the same time periods, you can run 5 10X more tests, or the same number of tests with far fewer engineers. Our customers report that their total cost of Ownership (TCO) is significantly less than with the freeware solutions they were using in the report. Load DynamiX improved our testing coverage and eliminated the need to buy additional servers. We saved significant dollars using your product. Terry Swartz QA Manager F5 Networks XII. Summary Nearly all of our customers used freeware tools like Iometer, Vdbench, and fio at one time, and they moved to Load DynamiX for the reasons explained in this paper. It s important to not lose sight of why these reasons are important. Better test and validation is key better decision making enabling our customers to: Evaluate the effect of new technologies like flash, OpenStack, and Ceph Evaluate which vendor or storage system performs best for each workload Pinpoint system resource needs / bottlenecks and optimal configurations, including tiering Conduct a more complete, accurate, and honest POC process Correctly stage to test the infrastructure before going live Pre-determine the effect of patches, upgrades and other changes; avoiding problems Load DynamiX. All rights reserved. LoadDynamiX.com 28