Why it matters and what you need to know
STORAGE EFFICIENCY Executive Summary In comparison with network-related parameters such as optimized TCP connections or WAN throughput, storage capacity is often overlooked when comparing WAN optimization products. Yet the amount of storage effectively available on a WAN optimization appliance is a key parameter for overall performance. This paper discusses the relationship between storage and performance, and shows how different architectures use storage more or less efficiently. It provides technical information that is important when evaluating and comparing WAN optimization products. Why storage efficiency is key to WAN optimization performance WAN optimization systems like Riverbed Steelhead products use disk-based storage to improve application performance and reduce bandwidth utilization. In a Steelhead product such as the Riverbed Steelhead appliance or the Steelhead Mobile software client, a disk-based data store is used to store segments of data coming from TCP flows that are optimized by the Steelhead product. The Steelhead appliance (or Steelhead Mobile software client) intercepts and analyzes TCP traffic, segmenting the data and indexing it. Once the data has been indexed, it is compared to data on the disk. If the data has never been seen before ("cold" transfer), the segments are compressed using a Lempel-Ziv (LZ) based algorithm and sent across the wide area network (WAN) to the counterpart device, which also stores them in its disk-based data store. If the data has been seen before ("warm" transfer), it is not transferred across the WAN; instead, a compact reference is sent to the counterpart, which uses this reference to look up the corresponding data in its data store and reconstruct the original data stream. By repeating this process over time, each Steelhead appliance gradually builds up a massive "dictionary" of data segments that can be leveraged to replace repetitions of those segments by references in subsequent transfers. Every time the Steelhead appliance (or Steelhead Mobile client) adds new data segments to its disk-based dictionary, it creates additional opportunities for eliminating redundant traffic. The data store fills up progressively over time, and eventually reaches full capacity. At that point, "old" segments must be removed before new segments can be stored. Naturally, it is desirable that the data store be large and efficiently used, so that the dictionary can contain the largest possible number of segments allowing more warm transfers and thus greater data reduction. Storage efficiency is an important consideration for WAN optimization systems. Efficiently designed systems can remember far more data than others with similar disk capacity resulting in greater scalability and performance in production deployments. There s more to it than raw capacity So does this mean that it is enough to compare the raw disk size of different products and infer that the one with the larger disk is better? Unfortunately not, because different product architectures have very different levels of storage efficiency -- meaning that for the same traffic, one system can consume disk storage at a vastly different rate than another system with a different architecture. In other words, it is not enough to simply compare disk capacity, or the data store size, of different WAN optimization products because a product that inefficiently uses storage may easily use its capacity ten or more times faster than a product using a storage-efficient approach. These varying degrees of storage efficiency arise from differences of product architecture in three areas: Universal vs. per-peer data store. A universal data store keeps all data segments in a single dictionary that is used to optimize traffic to all remote peer appliances and mobile clients. A per-peer data store uses a separate dictionary for each remote peer, thus storing multiple duplicate instances of data whenever the same data is accessed from different remote sites. De-duplicated vs. full data representation. With a de-duplicated data representation, data segments are stored a single time, even when they occur in different files and application transfers. With a full data representation (used by caches), each file or application object is stored in its entirety. Uni-directional vs. bi-directional data storage. With a bi-directional data store, data that is sent in one direction (e.g., from branch office to data center) is stored and re-used to optimize traffic going in the opposite direction. A uni-directional data store maintains separate storage areas for each direction of traffic, and so writes two redundant copies every time the same data is transferred from the data center to a branch office and back to the data center (or vice-versa). 2009 Riverbed Technology. All rights reserved. 1
The following sections describe each of these aspects in more detail, providing the background in storage architectures that customers need in order to ask the right questions when evaluating WAN optimization products. Universal vs. per-peer data store In a WAN optimization deployment, some appliances have multiple peers. For example, in a classic branch office deployment with a large data center appliance and smaller appliance in each branch office, the data center appliance will peer with each branch office appliance in order to optimize traffic to and from these offices. The branch office appliances may themselves also have multiple peers, for example when there are multiple core appliances are in the data center (for scalability and/or availability), when there are multiple data centers, or in a meshed topology where applications transfer traffic directly between branches. There are two possible approaches for structuring the data store of an appliance that has multiple peers: a per-peer structure, and a universal structure. Riverbed Steelhead products have a universal data store structure, while most other WAN optimization products have a per-peer structure. A per-peer data store allocates storage resources independently for each remote site, and stores a separate instance of data for each peer. A universal data store is shared among all peers. It does not store data separately for each peer, but rather stores a single instance of data, regardless of the number of peer associations with other WAN optimization endpoints. While data store organization is an internal, under the hood aspect of a WAN optimization system s architecture, it has important consequences on storage efficiency. Figure 1: A universal data store (left) keeps all data segments in a single dictionary. A per-peer data store (right) creates separate dictionaries for each peer, creating a scalability bottleneck as the number of remote sites increases. In this example, there are 8 remote peers. With more remote peers, the size of each peer partition with the per-peer data store would be even smaller. Per-peer storage inefficiency at the core As a first example of per-peer and universal storage efficiency, consider a file that is transferred from the data center to users in 10 branch offices. This file may for example be transferred as an email attachment, or retrieved by users via CIFS, or via a webbased document management system. With a universal data store, the byte patterns that make up the file data will be represented only once at the core appliance in the data center. The data transfer to every remote branch office leverages the same instance of data in the data center appliance. With a per-peer data store, the byte patterns must be redundantly stored 10 times in a core appliance a highly inefficient use of storage at the core of the WAN optimization infrastructure. Furthermore, this inefficiency will increase as the size of the deployment increases with 50, 100, or 1000 sites, the rate of consumption of the per-peer data store will be considerably faster. Per-peer storage inefficiency at the edge The impact of a universal vs per-peer data store is not restricted to core WAN optimization appliances. As a second example, consider a branch office appliance that connects to multiple core appliances. Those core appliances might all be in the same data center, or they might be spread across different data centers. Depending on the mechanism used to distribute traffic to the different core optimization appliances, each branch office appliance may end up optimizing the same traffic with different core appliances at different points in time. This could happen because multiple core appliances are used to optimize a given set of servers, with the choice of which core appliance to use being done on a per-client, per-session basis, and with different clients in a same branch being distributed to different core appliances. Or, even if core appliances are dedicated to optimization of different services (email, file, intranet), there are very likely similar or identical files being transferred via these services. In each of these situations, a branch office appliance with a per-peer data store will redundantly store multiple instances of the same data, and will consume its storage more rapidly than a universal data store. Over-provisioning is not the solution To overcome storage efficiency issues, vendors of per-peer systems may try to provision more appliances at the core than are 2009 Riverbed Technology. All rights reserved. 2
needed based on bandwidth or TCP connection sizing considerations. It is quite ironic that by doing so, the problem is shifted to branch office appliances, which will have more peers and thus greater per-peer inefficiencies to overcome. Therefore, even without considering cost, and even without considering the drawbacks of having more devices than necessary (e.g., management overhead, power consumption, hardware failures), customers should be wary of brute force over-provisioning approaches, which cannot fully overcome the shortcomings of a per-peer approach. Attempts to justify a per-peer data store Nearly all WAN optimization systems use a per-peer data store. Why would a vendor choose to use a per-peer data store, given the overhead and scaling limitations that come with this approach? While we cannot explain other vendors engineering decisions on their behalf, there are at least two reasonable observations that provide a useful context. First, a per-peer data store is simply the more obvious design. Second, a per-peer approach simplifies the problem of managing the conversation between each pair of communicating devices, and therefore easier to implement. When you dedicate resources to a particular peer, you have an easy time keeping track of which vocabulary to use in communicating with that peer. A universal data store avoids keeping separate vocabularies, which requires insight, ingenuity, and some additional work: but that effort pays off in much greater scalability. Taking both observations together, and considering that most other WAN optimization systems were designed and implemented in a catch-up mode with pressure to rapidly deliver a product to compete with Riverbed, it appears less surprising that most vendors use per-peer designs. The fallacy of per-peer protection Not surprisingly, WAN optimization vendors that use a per-peer data store have come up with various explanations that try to justify the inefficiencies of this approach. One claimed justification is that a per-peer data store provides protection from remote sites that transfer large amounts of traffic, by preventing that traffic from consuming excessive amounts of storage and thus penalizing other sites with less traffic. This claim confuses two issues that are entirely separate: the use of a per-peer vs universal data store, and the replacement policy used to evict old data from a data store. Regardless of architecture, any data store will inevitably fill up over time and require old data to be evicted. Choosing which data to evict is the task of a replacement policy, and a good replacement policy seeks to evict data that is least likely to be valuable in the future. This is necessary regardless of whether the data store is perpeer or universal the only difference is that there will usually be more replacements and faster churn in a per-peer data store, given its smaller per-peer partitions! Furthermore, a scenario such as the one above is equally possible with a per-peer data store. A single client doing a large transfer can evict data from the data store for that site, thus impacting optimization for all other clients at the site. And since each peer s portion of a per-peer data store is smaller than a single global data store, this situation is actually more likely to occur with a perpeer data store than a universal data store. Another way to shed light on the claim of per-peer protection is to consider the concrete example of web caches. With web caches, one could make the similar claim that cached objects should be stored separately (and redundantly) for each client or site in order to provide protection from misbehaving clients. Yet, to our best knowledge, there is no industry debate about restructuring web caches to implement per-client divisions. It is accepted and obvious that web caches use a universal data store. In summary, it is absurd to design a per-peer system that creates significant storage inefficiencies, thus reducing the effective data store size for each and every site, in order to confine the amount of storage available to high-traffic sites. A shared data store, by making more room on disk for all remote sites, will in most cases achieve the same effect without the downside of the per-peer approach. By using Riverbed pre-positioning capabilities, administrators can also ensure that transfers will be warm for critical data, at any chosen site(s). A full data store provides warm performance. So should we store redundant copies of data to fill it faster? One vendor has sometimes claimed that customers should not be concerned with how rapidly the data store fills up, because a full data store provides warm performance. (As it happens, that vendor has omitted from its management interface all information concerning disk usage, hiding disk storage utilization statistics from the administrator). Certainly, a full data store provides the warmest performance, but it is rather odd to suggest that this should justify storing the same data multiple times in order to fill it more rapidly! Taking this reasoning to its logical extreme reveals its absurdity: if rapidly reaching a full data store is an objective of 2009 Riverbed Technology. All rights reserved. 3
its own right, why simply not shrink it or write millions of redundant instances of the same data? Why worry? Disk is cheap! Yet another sometimes-heard refrain can be summarized as disk is cheap, so why should this matter? Certainly, unmanaged disk storage is inexpensive. If the only benefit of a universal data store were to save on a few hundred gigabytes or a few terabytes of storage, then it would be a minor optimization rather than a critical element of system architecture. However, this entirely misses the point at multiple levels. First, the storage efficiency afforded by universal data store is valuable for increasing the overall scalability of a WAN optimization device, rather than saving a fixed amount of storage. The amount of extra storage that would be necessary to compensate for the inefficiency of a per-peer data store will depend on many parameters, including the number of peers. As a deployment grows, so does amount of additional storage that would be needed to compensate for the per-peer inefficiency it is not simply a matter adding a known, fixed quantity of storage. As an extreme example, consider a deployment where strictly identical data is sent to all sites. This data could for example consist of software updates, system images, store catalogues, or video content. If we have a 100GB data store at the core, and a single remote site, then there is no difference between having a universal or a per-peer data store. If we have two remote sites, then the per-peer data store would need an extra 100GB of storage to keep up with the universal data store. That would probably be an affordable proposition, but what happens as deployment grows further? At 10 sites, we would need a 1TB per-peer data store, and at 100 sites we would need 10TBs suddenly making it harder to envision that throwing disks at the problem is a feasible way to solve it. Furthermore, adding unnecessary disk storage to a WAN optimization appliance increases its cost by far more than the mere hardware cost of the actual drives. All WAN optimization appliances must maintain some form of in-memory index of the byte patterns in their data store, and increasing the number of those byte patterns (even if they are redundant) will require an increase in costly RAM resources. Additional drives also require additional power and cooling a particularly important consideration in data center environments, where the overhead of per-peer is typically the greatest. Finally, disk may be cheap, but it also has high latency, so high-performance appliances sometimes use alternative technologies. For any products that use flash memory as a partial or total replacement for disk, the 'cheap disk' argument disappears entirely Not surprisingly, what vendors with per-peer architectures do not tell their customers is how it impacts the scalability of their solution. In a lab test or trial with a single remote site, there is no difference between a per-peer and a universal data store. With a handful of remote sites, the differences are usually still small enough not to have a significant impact on performance. However, as a deployment grows to 10, 50, or more remote sides, then the effects of a per-peer data store insidiously appear, resulting in a data store that fills increasingly rapidly and bandwidth reduction statistics that trend downwards as the deployment grows. De-duplicated vs. full data representation The per-peer data store issue described above is a key consideration for any WAN optimization appliance (or client) that connects with multiple peers. A separate consideration for storage efficiency is the way in which a single appliance (or client) represents data on disk; different approaches can summarily be described as using either a de-deduplicated representation or full data representation. There can be order-of-magnitude differences in storage overhead between both approaches. Furthermore, this overhead is independent of the per-peer vs. universal aspect; both are cumulative. Single-copy and caching architectures One form of multiple-instance storage comes as a direct consequence of using legacy caching-based architectures 1. In a caching based architecture, entire application objects are stored on disk. For example with a CIFS file cache, a file that is requested by a client will also be mirrored in the file cache. In a single-copy architecture such as the Riverbed Optimization System (RiOS ), application objects are not represented as such on disk. Rather than store an entire file (or other application object, e.g. email attachment or web page) in a filesystem, RiOS only writes the unique segments coming from that file. From a storage point of view, the RiOS single-copy approach is effectively de-duplicating data before writing it to disk and thus utilizing disk capacity far more efficiently than with the full representation employed by caches. This fundamental difference maps into substantial increases in effective storage capacity, as illustrated by the following examples: 1 Note that caching has several other drawbacks that are not discussed here for a complete overview of the problems of file caching, please refer to the Riverbed whitepaper There s No Free Lunch with Distributed Data Access 2009 Riverbed Technology. All rights reserved. 4
De-duplication across multiple copies of a file. It is common for identical files to be stored multiple times on a remote file server. For example, a user copies a file to a different folder in his share, or multiple users each save a copy of the same file to their individual shares. However many different copies of a file are accessed over the WAN, a single-copy architecture such as RiOS only consumes storage for one copy of that file, because all of the underlying segments are shared and reused as soon as they have been learned for the first transfer of that file. A cache-based architecture will separately store each copy that is accessed over the WAN. De-duplication across edited versions of a file. It is also common to find multiple edited versions of a file being sent back and forth across the WAN. For example, a user may edit and save several versions of a multi-megabyte PowerPoint file, with only a few modifications in each version. Because RiOS uses a fine-grained segmentation, with segments from eight to 512 bytes in length, it will only store segments corresponding to modified portions of an edited file. A cache-based architecture is oblivious to underlying commonalities between files, and must store each edited copy in its entirety. De-duplication across protocols. The previous examples are not specific to file protocols and file access over the WAN. The Riverbed single-copy approach also applies across protocols, so that the same data sent via different protocols need not be stored multiple times. Certain cache-based architectures not only do caching for file protocols, but also for HTTP, FTP, or Exchange attachments. If a user receives data as an email attachment, edits and saves it to a file share, and later uploads it into a web application, a multi-protocol cache will store three redundant copies of the this data, once again consuming storage resources more rapidly than RiOS-powered devices. Are there any situations where the RiOS single-copy approach would actually consume storage faster than a cache-based system? The answer is no in the least favorable case, a single-copy approach would use the same amount of storage as a cache, but never more. This would occur in the unlikely situation that all data traversing the WAN optimization appliance is noncompressible (e.g. media files or compressed files), and every user in a given remote site accesses entirely different sets of data and files. Hybrid architectures Over time, some vendors of caching systems have introduced byte-level data reduction mechanisms imitating Riverbed in an attempt to capture the benefits of the Riverbed approach. These products keep their legacy caching mechanisms, but add a bytelevel data store operating underneath. In such hybrid architectures, accelerated data is stored first as entire application objects (e.g. files, HTTP objects, email attachments) in a cache, and then a second time as byte-level data patterns in a data store. While the addition of a byte-level data store can improve the performance of a cache for data reduction on the network, it makes things even worse when it comes to storage efficiency. These hybrid systems use even more storage than a pure cache-based system, because they consume the same amount of storage for the cache, and additionally, use disk space to maintain a dictionary of data segments. 2009 Riverbed Technology. All rights reserved. 5
Location A Location B Bi-directional Data Store Bi-directional Data Store Uni-directional Store data from A to B Uni-directional Store data from A to B Uni-directional Store data from B to A Uni-directional Store data from B to A Figure 2: A bi-directional data store (top) keeps all data segments in a single dictionary, independently of the direction of traffic. A uni-directional data store (bottom) uses separate dictionaries for each direction of traffic, thus storing duplicate data segments and using storage less efficiently. Uni-directional vs. bi-directional data store The third aspect of storage efficiency affects both core and edge devices. A uni-directional data store has separate storage areas for each direction of traffic. In other words, a uni-directional data store maintains one dictionary for data going from the LAN to the WAN, and another dictionary for data going from the WAN to the LAN. In contrast, a bi-directional data store has a single dictionary that stores data once, regardless of traffic direction. Riverbed Steelhead products use a bi-directional data store, thus avoiding the inefficiency of a uni-directional approach. Each time the same data traverses a uni-directional data store in both directions, this data will be redundantly stored twice, thus consuming storage two times faster than a bi-directional data store. Such back-and-forth traversal of data on the WAN is common in enterprise traffic, for example when a client reads a file from a remote share, edits it, and saves it. Another frequent occurrence is when a client receives an email attachment or downloads a document via HTTP, and subsequently saves it to a remote share. Suggested questions for WAN optimization vendors As this document shows, there are many different aspects to storage efficiency. This section lists some questions that prospective customers of WAN optimization products can ask in order to assess the storage efficiency of different vendors architectures. 1. Does your product use a single disk-based dictionary (universal data store) for all remote peers, or does it use one separate dictionary (per-peer data store) for each remote peer? 2. In order to use the full storage capacity of edge appliances and software clients, is it necessary for the data center appliance(s) to have storage capacity equal to the total of all remote storage? For example, if there are 50 remote sites equipped with appliances with a 30GB data store, does the data center require appliance(s) with 1.5TB of data store in order to make full use of the remote appliances data stores? 3. Does your product store application objects (e.g. files, emails, or attachments) in their full representation, or does it only store unique data segments coming from these objects? 2009 Riverbed Technology. All rights reserved. 6
4. If I make 10 copies of a same file (e.g., file1.ppt, file2.ppt,.. file10.ppt) on a file server, and access them over the WAN, will the client-side appliance (or software client) consume 10 times the amount of storage that it would use if I only accessed a single copy of that file? 5. If the same data is transferred in one direction (e.g., from file server to edge), and then in the opposite direction, will your product consume more storage than if it was only sent one way? 6. Do the data sheets for your products distinguish between raw disk size and the amount of disk space that is available for the data store? If not, what is the disk space that available to the data store? 7. Does the monitoring interface show how much of the data store is currently used? Can the system be configured to raise an alarm if the data store fills up more rapidly than expected? Conclusion Storage efficiency is critical to the real-world performance of WAN optimization products, and the importance of storage efficiency increases with the number of sites, the volume of traffic, and the number of different kinds of traffic optimized. Accordingly, storage efficiency should be evaluated by any prospective customer when comparing WAN optimization products. Riverbed Steelhead products were designed with storage efficiency as a primary concern. As a result, the Riverbed storage architecture uses a universal data store, de-duplicated data representation, and a bi-directional data store. Other products use a per-peer data store, full-blown data representation, and/or uni-directional data stores. The net result of such inefficient architectures is that storage space is consumed at much higher rates than with Riverbed products, seriously reducing the scalability and performance of these solutions. About Riverbed Riverbed Technology is the IT infrastructure performance company. The Riverbed family of wide area network (WAN) optimization solutions liberates businesses from common IT constraints by increasing application performance, enabling consolidation, and providing enterprise-wide network and application visibility all while eliminating the need to increase bandwidth, storage or servers. Thousands of companies with distributed operations use Riverbed to make their IT infrastructure faster, less expensive and more responsive. Additional information about Riverbed (NASDAQ: RVBD) is available at www.riverbed.com Riverbed Technology, Inc. 199 Fremont Street San Francisco, CA 94105 Tel: (415) 247-8800 www.riverbed.com Riverbed Technology Ltd. 1, The Courtyard, Eastern Rd. Bracknell, Berkshire RG12 2XB United Kingdom Tel: +44 1344 354910 Riverbed Technology Pte. Ltd. 391A Orchard Road #22-06/10 Ngee Ann City Tower A Singapore 238873 Tel: +65 6508-7400 Riverbed Technology K.K. Shiba-Koen Plaza Building 9F 3-6-9, Shiba, Minato-ku Tokyo, Japan 105-0014 Tel: +81 3 5419 1990 2009 Riverbed Technology. All rights reserved. 7