Applied Technology Abstract Navisphere Quality of Service Manager provides quality-of-service capabilities for CLARiiON storage systems. This white paper discusses the architecture of NQM and methods for maximizing its benefits. Sample use cases for NQM are also detailed. October 2006
Copyright 2006 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com All other trademarks used herein are the property of their respective owners. Part Number H2455 Applied Technology 2
Table of Contents Executive summary...4 Introduction...4 Audience... 5 How it works... 5 Running multiple policies... 6 NQM algorithms...7 Cruise Control... 7 When to use... 7 Limits... 7 When to use... 8 Fixed Queue Depth (for experts only)... 8 Replication application integration...9 Considerations and performance overhead...9 Performance overhead... 10 Example use cases...10 Monitoring application performance to discover bottlenecks... 10 Prioritizing applications during peak hours of operation... 11 Mission-critical application performance... 12 Conclusion...12 Appendix: NQM quick reference...13 NQM value propositions... 13 NQM facts and limits... 13 Applied Technology 3
Executive summary Quality-of-service (QoS) tools give users the ability to meet specific performance targets for their applications by allocating system resources accordingly, providing a level of control beyond that of any midrange storage-system. QoS extends the Information Lifecycle Management (ILM) strategy to include tiering application performance within a storage system. This makes QoS a valuable addition to any storage environment, especially those where high service level requirements remain for certain applications. A QoS tool should meet a minimum set of requirements to be effective. When performance is an important consideration for one or more applications sharing a storage system, there are two main requirements that must be addressed. First, there is a need to understand the true performance of applications at the storagesystem level. Monitoring capabilities must be provided by a QoS tool to help isolate the issue and help determine whether the bottleneck is at the storage-system level. Second, a QoS tool needs to prioritize certain applications because they are mission-critical. How do we measure effectiveness of a QoS tool? By delivering performance with a cost-effective configuration. When one or more mission-critical applications cannot meet their desired service levels, the solution without a QoS tool is to overprovision, which can increase costs and cause a temporary but significant performance impact on a production system.. Navisphere Quality of Service Manager (NQM) addresses these problems by enabling the dynamic allocation of system resources to meet service level requirements. NQM gives users the ability to prioritize applications and set specific performance targets so that users get the service levels they would like to obtain for their mission-critical applications. NQM also has the ability to monitor system performance on an application-by-application basis, giving a logical view of application performance on the storage system itself. Introduction NQM measures, monitors, and controls application performance on the EMC CLARiiON storage system. The monitoring feature is an excellent first step when attempting to improve performance of highpriority applications, because it gives users a more logical view of system performance, both for the entire storage system and for specific applications. This can be a powerful method of evaluating the storage system to determine the current service levels and to provide guidance on what service levels are possible, given the specific environment. NQM controls application performance by giving users the ability to set performance targets for highpriority applications or performance limits for low-priority applications. These performance targets and limits are one of three key performance characteristics response time, bandwidth, and throughput. For example, an email application running on a CLARiiON storage system can be given a specific response time performance target to achieve (on the storage system itself). Similarly, a noncritical application can have its bandwidth limited to a certain level so that the storage system has resources available for more important requests. In addition, NQM has a built-in scheduling feature so that these performance targets can be adjusted dynamically based on business needs. The scheduler allows users to give priority to certain applications during specified windows. For example, a database application can be given a performance target during the day, during its peak hours of operation. Overnight, it is critical that this important information be backed up in a timely fashion, so NQM is used to create another performance target for the backup application during that time period. Both of these can be scheduled to run automatically. NQM is fully integrated into the Navisphere user interface as shown in Figure 1 and has easy-to-use wizards that guide you through monitoring and setting goals in your storage system. Navisphere Secure CLI also provides full support to use NQM in a scripted, command-line-based environment. This white paper will discuss how NQM works, its algorithms, and some considerations to be aware of. Several Example use cases for NQM are also reviewed. Applied Technology 4
Figure 1. Navisphere integration of NQM Audience This white paper is intended for any EMC customer, partner, or employee who wants more information on how NQM works and how to implement it in a customer environment. This paper assumes the reader is knowledgeable in modular storage-system performance, CLARiiON performance characteristics, and performance tuning methods. Information on CLARiiON performance may be found in the EMC CLARiiON Best Practices for Fibre Channel Storage white paper on EMC Powerlink. How it works NQM is a storage system-resident tool. It runs completely on the CLARiiON storage system; there are no host-based components to install. Enabling NQM is as simple as installing the license key. Another important aspect of NQM running on the storage system is its performance measurements. NQM measures and reports performance characteristics on the storage system. Actual application performance will depend on several factors that lie outside of the CLARiiON, including the SAN infrastructure, host, and application. The NQM configuration process consists of five steps: 1. Create I/O classes. Applied Technology 5
2. Monitor application performance. 3. Set goals or limits. 4. Create and schedule policies. 5. Run NQM. Let s look at these steps in detail. Creating I/O classes The storage system is not, per se, application aware. I/O classes are the proxy for application profiles on the system. These classes can specify I/O requests on an application-by-application basis (by selecting specific LUNs or MetaLUNs), by I/O size (over 2 MB), and by I/O type (read or write). Monitoring application performance on the storage system Once the I/O Classes have been created, NQM is ready to monitor overall storage-system performance as well as individual I/O class performance. This step is essential in determining current application service levels on the storage system and how applications are performing in relation to their service level requirements and the overall storage system. Setting goals or limits After I/O classes are defined and application performance monitored, service goals are set for each I/O class. NQM measures three key application performance characteristics throughput (IOPS), bandwidth (MB/s), and response time (ms) for which goals may be set. The service goal should be both reasonable and the appropriate performance characteristic to match the application. For example, OLTP applications are typically measured in throughput or response time, while backup applications are measured in bandwidth. Along with specifying a performance goal, the user must choose a control method, which identifies how NQM enforces the service goals defined for the I/O class. There are two major types of control methods: performance targets and limits. Performance targets are specific targets for NQM to achieve for one of its I/O Classes. This method allows a user to allocate resources to a specific high-priority application. Limits are a way to put rules in place to ensure an application will not exceed a certain level. Limits allow a user to throttle back the amount of resources given to a low-priority application, which frees up resources for mission-critical applications. The combination of defined service level goals with a control method provides a set of instructions for NQM to follow. For example, an email application currently running at a 40 ms response time may be given a performance target of 20 ms. Creating policies Next, an NQM policy is created. A policy is a group of I/O classes for which service level goals are enforced using a certain control method. All I/O classes within a policy must use the same control method. For example, a policy may limit the throughput of an email application and the bandwidth of a backup application. Optional settings for a policy include the time limit for achieving those service goals and an action to take if NQM fails to achieve those goals within the time limit. What happens to all of the I/O from other applications that is not specified in any of our I/O classes? Those are automatically included in the policy via the background I/O class. The background I/O class can not be assigned service levels or a control method, but it allows the user to see how the rest of the system is affected by enforcing service goals on our defined applications. Running multiple policies Since each policy controls every I/O coming in or out of the storage system, NQM will not run more than one policy at a time. However, different policies may be scheduled up to a maximum of 10 policies to be enforced at different times, so that users can set different performance targets for the same application based on the time of day. For example, Exchange may be overused from 9 A.M. to 10 A.M. when people Applied Technology 6
come in and check their email. A policy for Exchange may be created that defines a performance target of 10 ms and is scheduled to be enforced during that time period. From 12 A.M. 2 A.M., a backup application runs on the same storage system. A second policy for Exchange may be created that defines a limit of 20 ms and is scheduled to be enforced during that time period. This throttles back performance given to Exchange during the backup window, which frees up resources for the backup application. Another important feature of NQM s scheduling capabilities is the optional fallback policy. This is the default policy that NQM will run, if no other policies are running. NQM algorithms NQM adjusts application performance by measuring I/O characteristics and prioritizing I/O within the storage system to achieve defined service level goals. This functionality is implemented by three control methods Cruise Control, Limits, and Fixed Queue Depth. Each control method implements a different algorithm and offers a different approach for achieving desired service level goals. Cruise Control The NQM Cruise Control method is analogous to the cruise control feature found on many automobiles. In NQM, a specific performance target is defined for a mission-critical application and NQM will prioritize I/O accordingly such that the application s performance meets the defined goal, within a specified tolerance range. In an automobile, a cruising speed is set and correct amount of gas is allowed into the engine until the car reaches the defined speed. NQM does not have the ability to add resources to the storage system in the manner that a car can add gas to the engine. Instead NQM must manipulate the available resources to achieve its goal, which is done by slowing down I/O to other applications so that the critical application can process additional I/O. How does NQM know by how much to slow down the background class? NQM will slow down the background class using various rates and measure the impact on the critical application s performance. The system will learn which values work well and readjust the rates to gather additional data points. This sampling and learning period is repeated until NQM finds the optimal settings by which the critical application will maintain its defined performance target. Cruise Control will also remember these optimal settings when the same policy is executed in the future. This intelligence allows Cruise Control to attain the desired performance goal faster on subsequent runs of a policy. The Cruise Control method also dynamically adjusts to workload changes, similar to how an automobile s cruise control feature will adjust the amount of gas to the engine if the car s speed begins to decrease. When NQM recognizes a prolonged workload change, the algorithm will adjust the I/O priorities so that more I/O is serviced for the critical application until its performance falls within the defined tolerance range again. In the automobile example, a prolonged workload change is analogous to the car going up a hill and starting to lose speed. In this scenario, more gas is allowed into the engine until the car regains its set cruising speed. When to use Cruise Control is best applied when only one or two mission-critical applications need to maintain a certain service level. For example, a backup must finish in a certain time window but there are other applications preventing this from happening because they are sharing the same resources. Due to the complex nature of the sampling and learning period, the Cruise Control method can be applied only to two I/O classes at one time. Limits The Limits method behaves as the name implies it limits the performance of an I/O class to a certain level. NQM will queue I/O requests for the I/O class to keep its performance under the defined limit. Referring back to the car analogy, the Limits method acts as a governor on an I/O class performance. A Applied Technology 7
governor on an engine limits the speed of the engine by controlling the rate of fuel delivery. The amount of fuel allowed into the engine never exceeds the governor s limit, regardless of how hard the driver presses on the gas pedal. By applying a limit to an I/O class, other applications that share the same resources will see a performance increase, as more resources are available for their I/O requests. While the Limits method tries to stay below the limit it also tries to stay as close to the limit as possible without exceeding it; thereby maximizing the application s performance within its limit. When to use The Limits method is best applied to noncritical applications that contend for resources with critical applications. As the limits are enforced and satisfied, resources are freed up for use by the critical application, thus boosting its performance. Some use cases for limits include enforcing SLAs for example, limit the bandwidth usage of a certain application and apportioning performance for Fibre Channel vs. iscsi applications for example, limit the iscsi applications so that they do not absorb most of the storage-system performance and negatively impact the FC applications. Fixed Queue Depth (for experts only) The first two control methods use algorithms to determine the appropriate I/O priorities needed to satisfy a service goal. The Fixed Queue Depth control method provides direct control of the amount of I/O processed for an I/O class. This method keeps the rate of I/O serviced constant for the I/O class and does not adapt to workload changes. As such, this method should be considered an advanced feature and should be used only by expert users with deep storage performance knowledge who understand the consequences of setting the I/O service rate of an I/O class. Applied Technology 8
Replication application integration NQM is designed to support user LUNs only; it does not support control of private LUNs. With respect to replication NQM does support the CLARiiON add-on applications. Table 1 lists the types of LUNs that NQM does and does not control for each of the replication applications. Table 1. LUN support Replication application May be NQM controlled May NOT be NQM controlled SnapView Source LUN Snaphot LUN (future support) Reserved LUN Clones Source LUN Clone Private LUN Clone LUN MirrorView /S Primary Image Secondary Image (can be put in I/O class but cannot be controlled unless promoted) Write Intent Log MirrorView/A Primary Image Reserved LUN Reserved Snapshot LUN Secondary Image (can be put in I/O class but cannot be controlled unless promoted) SAN Copy Source LUN Destination LUN (in case of back-end copy it can only put in an I/O class but NOT controlled) Destination LUN (in case of front-end copy it can be controlled from the remote storage system) Inc. SAN Copy Source LUN Destination LUN (in case of front-end copy it can be controlled from the remote array) Destination LUN (can be put in I/O class in case of back-end copy) Reserved LUN Considerations and performance overhead Storage-system performance is a key component of any application. As NQM directly impacts application performance on the storage system, there are several items that must be taken into consideration in order to properly configure and derive maximum value from both NQM and the storage system. NQM does NOT boost overall storage-system performance. Performance resources in any storage system is finite (unless you purchase more hardware). Therefore, as more performance resources are allocated to one application, performance will be taken from another application. NQM attempts to optimally allocate performance resources among applications, while maintaining service level goals identified by the customer. The Cruise Control method may take some time to achieve its goal. The sampling and learning period adds some time to achieve goals when compared to the Limits method. However, a Cruise Control feature remembers the last solution that was found and applies it when the same policy is subsequently run. In essence, the sampling and learning period is executed only the first time each Cruise Control policy is executed. Set reasonable goals. A policy is executed for a specified time limit and then an action is taken if the policy is not satisfied. Also keep in mind that as performance increases for a critical application, Applied Technology 9
performance may decrease for other applications. If service goals are too aggressive NQM might not be able to achieve them and noncritical applications may be starved of performance. Only one active policy is allowed. NQM allows for up to 10 policies to be scheduled, but only one is allowed to be run at any given time. All I/O classes in a policy must have the same control method. Currently, it is not possible to mix control methods within a policy. So, a policy can not have one I/O class with a Limits policy and a second I/O class with a Cruise Control policy. Performance overhead As with any add-on module to a CLARiiON, NQM increases the overhead of the storage system. This increase is a result of an increase in the storage processor s CPU utilization when NQM is running. Note that this overhead affects the application only if the storage system is actually the bottleneck. In addition, the impact of the overhead is less when the disk drives are the bottleneck in the storage system, since most of the I/O service time is spent at the disk and higher CPU utilization will not introduce that much more additional latency into the I/O path. The results presented below are based on testing with CX300, CX500 and CX700 storage systems. The overhead for CX3 Series storage systems is expected to be less since the overhead is CPU bound and the CX3 Series offer a significant increase in CPU power. The performance overhead for NQM represents the impact on I/O performance (IOPS, BW, RT) and not just CPU impact. NQM installed vs. NQM not installed: NO IMPACT NQM idle vs. NQM measuring: ~5% NQM idle vs. NQM running: As NQM dynamically adjusts the performance of individual applications when a policy is executing, it is not practical to calculate the performance overhead of NQM. The performance of the critical application(s) is more pertinent than the maximum aggregate performance of the storage system. However, it is expected that NQM s performance overhead is only slightly higher when running a policy than when measuring a policy. The results indicate that NQM has a minor impact on the CLARiiON s aggregate performance when NQM is actively measuring or running a policy. Example use cases NQM has several possible uses from monitoring application performance to increasing mission-critical application performance. The use cases below are representative of possible uses of NQM. Monitoring application performance to discover bottlenecks As computing environments become more complex, it is increasingly difficult to isolate application performance on the storage system itself. This makes it difficult to determine the source of the problem if end users are experiencing poor performance with their applications. Poor application performance depends on several factors including server workload, application configuration, network communications, or storage workload. NQM s built-in monitoring tools allow users to monitor the performance of applications on the storage system. For example, users of an Oracle database application are complaining of poor response times. Using NQM in measuring mode will allow the customer to determine if the storage system is the cause of the performance problem, as shown in Figure 2. If the end-user response times are significantly higher than those reported by NQM, this is an indication that the performance issue lies outside of the storage system. Applied Technology 10
Figure 2. NQM monitoring features Prioritizing applications during peak hours of operation Users are also constantly being asked to adjust to workload changes in their environment and adapt to multiple service level requirements for different applications during different times. This can be a challenge to maintain and all the man-hours devoted to the task can be costly. NQM s scheduling functionality allows you to create different service level goals throughout the day, so you can set performance targets for certain applications during their peak usage. Let s consider the following example: A company has an Exchange server and a backup application residing on the same CLARiiON storage system. They want to ensure that the Exchange environment meets its throughput target during the day (7 A.M. - 8 P.M.). At the same time, they want the backup application to complete its nightly job during the current backup window (11 P.M. 3 A.M.). To accomplish this, the customer creates two separate policies, one for Exchange and one for the backup application, and schedules each policy to run at the appropriate time, as shown in Figure 3. Applied Technology 11
Figure 3. Scheduling policies Mission-critical application performance As storage-system capacities continue to increase, consolidation of multiple high-priority applications on to the same storage system is also increasing. There is a need to ensure performance on true mission-critical applications running on those storage systems. NQM allows a customer to set limits on low-priority applications, for example, like iscsi hosts connected to an EMC CLARiiON CX3 UltraScale system with Fibre Channel and iscsi connectivity. This will ensure that the storage array has dedicated resources available for the high-priority applications, like those connected via FC. Conclusion Information Lifecycle Management (ILM) is traditionally focused on tiering storage placing data on the correct storage in terms of price and performance throughout its life cycle. Storage consolidation has yielded situations where multiple high-tier storage applications contend with low-tier storage applications for resources. Previously, the method used to alleviate this contention was to reallocate applications to new disks in the storage system or an entirely different storage system. NQM offers CLARiiON users a new slant on ILM the ability to tier application performance on the array. Now storage-system resources are adjusted such that critical applications achieve performance goals without the need to buy new hardware or reprovision the storage system. NQM controls application performance levels by continuously and dynamically measuring performance against defined service level goals. The NQM monitoring functionality gives the customer an application-centric view of performance. The integrated scheduler lets a customer dynamically adjust which applications are considered the high-priority applications, based on business needs. All measured performance statistics are also automatically archived on the storage system. Applied Technology 12
Appendix: NQM quick reference NQM value propositions Monitor and achieve performance objectives for applications. Schedule user-defined policies that allocate system resources dynamically to meet service levels. Benefit from consolidation while optimizing the performance of mission-critical applications. NQM facts and limits NQM is an optional add-on feature of the Navisphere Management Suite that requires a separate software license. Installation of the NQM enabler file is a rebootless process. NQM is available for the CX3 Series storage systems and the CX300, CX500, and CX700 models running FLARE Release 24 or later. NQM is a completely independent product from Navisphere Analyzer; however, both applications can operate on the same storage system. An I/O class may contain a maximum of 64 LUNs. Policies have the following limits for each control method: Cruise Control has a maximum of two I/O classes per policy. Limits has a maximum of 32 I/O classes per policy. A maximum of 10 policies may be scheduled in NQM. NQM automatically archives up to seven days worth of performance data. NQM archives may be retrieved and stored on a local server for later use. Archives have a.nqm file extension and must be viewed with the NQM Archive Viewer. Applied Technology 13