Study and Performance Analysis of IP-based Storage System

Study and Performance Analysis of IP-based Storage System Zhang Jizheng, Yang Bo, Guo Jing, Jia Huibo Optical Memory National Engineering Research Center (OMNERC), Tsinghua University, Beijing, China 100084 ABSTRACT Storage technology has enjoyed considerable growth since the first disk drive was introduced nearly 50 years ago. Various storage devices with large volume and high performance are emerging and they carry the important information for our social life. How to make the devices more broadly and effectively used and make the information more available is a problem for today. IP-based Storage provides such a cheap interface for storage devices under the rapid development of network technology. This paper answers what is network storage relative to traditional storage and then puts emphasis on two typical IP-based storage technologies and iscsi. The article evaluates the performance of and iscsi by measuring and analyzing file-level access performance and block-level access performance. Our test results show that iscsi provides higher performance then using NFS protocol in Linux or SMB protocol in Windows ( scheme). The last part of the paper gives characteristic and trade-off of several IP-based storage system architectures. Keywords: Network Storage,, iscsi, SAN 1. INTRODUCTION Today, exponential data growth requires data storage systems that are massive in terms of storage capacity and access performance. As the foundation of storage system, storage devices develop rapidly. Since the first disk drive in 1956, disks have grown by over six orders of magnitude in density and over four orders in performance. How to make the devices well used to satisfy user needs becomes a problem for today. Modern storage system should consolidate resources, allow data sharing, be possible to distribute them over global distances and at the same time be centrally managed. It should also deploy quickly, be highly available, make them secure against external and internal abuse, and scale their performance with capacity. It is undoubted that storage plays more and more important role nowadays. Data storage was previously thought to belong to the inside of server cases or tight next to a mainframe computer. However, as networking technology started to affect storage device connections, the placement and distance of storage systems gradually changed. First, they moved to their own storage subsystems; then, they were located at almost any distance from the computers they serve. Formerly known as peripheral devices, storage systems became the center of many Information Technology (IT) operations. Today, data storage systems constitute major portions of many IT budgets. In data centers, computing nodes (servers) became peripheral to the storage systems; servers are upgraded and connected to storage systems to satisfy the data processing requirements. Storage becomes an independent storage subsystem, departs from host and is networked as a network node. We call it network storage (also known as storage networking or networked storage ). The rapid development of networking technology is the base of network storage. And at one time, two mostly uncorrelated computer technology areas, networking and storage, now seem to be inseparable. Network storage is considered to have the same significance as Personal Computer (PC) and Networking before in Information Technology. Traditional storage includes local storage and storage within C/S (/) model. Both of them have the same characteristic that they connect block-based storage devices directly to the I/O bus of a host machine (e.g., via SCSI or ATA/IDE). As illustrated in Fig. 1, storage devices are directly connected to client in local storage. Though in C/S storage, client accesses server s data through network (LAN in Fig. 1), storage devices are still directly connected to the server. In each case, storage system is peripheral of the host. This kind of storage is called DAS (Direct Attached Storage) or sometimes Host Attached Storage. 150 Advances in Optical Data Storage Technology, edited by Duanyi Xu, Kees A. Schouhamer Immink, Keiji Shono, Proceedings of SPIE Vol. 5643 (SPIE, Bellingham, WA, 2005) 0277-786X/05/$15 doi: 10.1117/12.571020

iscsi Device LAN SCSI ATA/IDE SAN Fig. 1. Simple introduction of DAS, and SAN (Network Attached Storage) and SAN (Storage Area Network) are two of the most important technologies of network storage. As illustrated in Fig. 1, specialized file server that serves file system data over a network is generally referred to as server (or directly ). We can consider as an independent storage device directly connected to LAN. SAN is a specialized network for storage data that can connect multiple hosts to multiple storage devices. Any high speed networking technologies can be used in SAN such as FC (Fibre Channel), GE (Gigabit ), Infiniband, etc. iscsi (Internet SCSI) is the upper protocol used in IP SAN (e.g., iscsi/tcp/ip/). FC, iscsi and Infiniband are all transport mechanisms to transfer SCSI protocol data. iscsi is an IP-based technology, so iscsi can be widely used not only in SAN but in LAN and even WAN. and iscsi are both IP-based storage technologies. As a general communication infrastructure, IP-based technologies have experienced a widespread success that no other data communication technology, before and after it, has. is another success story that made this technology the de facto standard for local area networks (LANs). With widespread use come economies of scale and reduction of costs. In addition, kept increasing its speed 10- fold in every generation. These two trends provide IP/-based networks a cost/performance advantage that cannot be matched by any other data communication technology. Because of IP/ s cost/performance advantage and widespread availability, it appealed to storage professionals as a means to construct and supplement storage networks. IP-based storage, storage virtualization and /SAN combination are considered as the development trends of network storage. The remainder of this article is organized as follows. We introduce and iscsi in detail, draw a comparison between the two IP-based storage technologies and evaluate the performance of them by measuring and analyzing. We discuss today s prominent storage architectures including /SAN combination and the trade-off involved. At last, we design and implement an IP-based storage system for data center and conduct real performance measurements for it. 2. AND ISCSI and iscsi are both IP-based storage technologies. If we use iscsi devices directly in data network such as LAN instead of storage network, devices and iscsi devices are almost the same in their physical connections. But they are essentially different. 2.1. Proc. of SPIE Vol. 5643 151

A device is a special-purpose storage system that is accessed remotely over a data network. s access via a remote-procedure-call (RPC) interface such as NFS (Network File System) for UNIX systems, or CIFS (Common Internet File Share, also known as SMB, Message Block) for Windows machines. The remote procedure calls are carried via TCP or UDP over an IP network -- usually the same LAN that carries all data traffic to the clients. File Web Mail SQL... General Web Mail SQL... General File File General File Data Other Data (a) (b) (c) Fig. 2. Network Attached Storage is a reduced specialized server only used for file service (Fig. 2b). device is usually implemented as a RAID array with software that implements the remote procedure call interface. It is easiest to think of as simply network storage device such as a disk (Fig. 2b). 2.2. iscsi Currently, the most widely used storage access protocol is SCSI. The SCSI protocol supports several underlying I/O interconnects. The parallel SCSI bus is the most widely used interconnect between SCSI devices and a host. However, parallel SCSI has several limitations. The fundamental limitations are distance and the number of storage devices that can be attached. A parallel SCSI bus can only stretch to several meters and attach at most 16 disk devices. These limitations restrict the scalability of a storage system. Initiator Target SCSI iscsi TCP IP Link SCSI iscsi TCP IP Link IP Network Fig. 3. iscsi protocol layering model Fibre channel has long dominated the realm of SAN. However with prevalent deployment and accessibility, IP network can be exploited to serve as storage data carriers to extend the storage system. Emerging as an end-to-end protocol for transporting storage I/O block data over IP networks, iscsi encapsulates disk access requests (in the form of SCSI CDB 152 Proc. of SPIE Vol. 5643

commands) into TCP packets, and transmits the SCSI commands and block-level data over IP networks (Fig. 3). Thus, iscsi encompasses two major protocols: for storage access, SCSI, and for networking transport, TCP/IP. By exploiting the ubiquitous Internet infrastructure, it extends the SAN network to a remote area and greatly facilitates remote storage, remote backup, data mirroring, and remote management. It also unifies the storage and data networks, thus greatly reducing management cost. As shown in Figure 3, the iscsi layer interfaces to the operating systems standard SCSI set. The iscsi layer includes encapsulated SCSI commands, data and status reporting capability. The iscsi protocol monitors the block data transfer and validates completion of the I/O operation. This occurs over one or more TCP connections between initiator and target. In practical applications, an initiator may have multiple target resources over an IP network, and consequently multiple concurrent TCP connections active. SCSI commands are typically issued by a storage initiator (client) to a storage target (server). iscsi initiator is generally host and iscsi target is networked iscsi device. There are three options for iscsi initiator implementation (Fig. 4). The easiest solution is to put iscsi software drivers in initiator computers and handle all processing using the host CPUs. Although this is a low-cost solution, it will slow down the host processor. To improve the performance, we can use a Network Interface Card (NIC) with TCP Off-load Engine (TOE) or an iscsi Host Bus Adapter (HBA) off-loading not only TCP but iscsi protocol. Fig. 4. Three options for iscsi initiator implementation 2.3. Comparison between and iscsi The essential difference between two IP-based storage technologies and iscsi is illustrated in Fig. 5. Since NFS is a representative protocol of, we use it to explain the difference. The host application initiates the operation by a regular system call. If the operating system identifies the file as a local one, the operation is given to Local File System. If the file is identified as a remote one, the operating system will invoke the appropriate NFS procedure. An RPC call is made to the NFS service layer at the. The call is reinjected to the system, which finds that it is local and invokes the appropriate file system operation. Let s go back to see the first case that the operation is given to Local File System. Then the operation is mapped to storage device. If the operating system identifies the device as local disks, the operation will be finished as local storage (DAS). If the storage device is identified as remote iscsi device, the operating system uses iscsi protocol to establish one or more sessions between the host and the iscsi device and perform the operation in iscsi device. Proc. of SPIE Vol. 5643 153

Application Local File System Volume Manager Disk I/O System iscsi Initiator Block I/O IP Network iscsi Target NFS File I/O IP Network NFS File System Volume Manager Disk I/O System Block I/O Local Disks Disks Disks iscsi Device Device Fig. 5. Comparison between and iscsi uses high-level file I/O access protocols while iscsi is a relatively low-level block I/O access protocol. iscsi can be used not only in file service but also in database because upper applications have no influence to iscsi. We can consider iscsi device as remote disk that has the same usage as local one. We test the performance of and iscsi. In order to better explain the difference between them, we use the same hardware to construct device and iscsi device. The testbed environment is described as follows. CPU Celeron 1.7G Memory 256M (DDR266) Disk IBM DDYS-T18350N (18G) SCSI Disk Network Interface 3COM 3C996B-T 1000Base-T Adapter OS RedHat Linux 7.2 Kernel (2.4.7) Tab. 1. Component of and iscsi device We use the same SCSI disk here for two devices even though ATA/IDE disk is also available for. Gigabit NIC is directly connected to a host s Gigabit NIC without passing through a Gigabit Switch for each device to eliminate the influence of Switch. and iscsi are both implemented by software on Linux. We use IOMeter 2003.05.10 as the benchmark to compare iscsi with SMB ( scheme in Windows) protocol and Bonnie++ 1.03a to compare iscsi with NFS ( scheme in Linux) protocol. It is obvious that local storage with SCSI disk should have the highest performance in the test, and its throughput represents the upper limit of the SCSI disk. As shown in Fig. 6, the throughput of iscsi is close to local SCSI when read and almost the same when write. That means iscsi disk can be considered as a local SCSI disk if ignoring the iscsi and TCP/IP protocol overhead. Whereas the performance of is far lower than them. The results show that iscsi provides higher performance than with SMB protocol from one client s point of view. When we increase the number of clients, total throughput of will increase and finally achieve the limit no less than iscsi. On the contrary, when the request size is big enough, one client can even lead the iscsi disk close to its limit (Fig. 6), so more clients won t help more increase iscsi disk s total throughput. However a user doesn t care the total throughput but the 154 Proc. of SPIE Vol. 5643

throughput he can get, so block-based iscsi with lower protocol overhead and higher performance than is a good choice for him when there are not many other hosts using the iscsi disk. 35 30 Read 20 18 16 Write Throughput (MB/s) 25 20 15 10 5 SCSI iscsi 0 0 512B 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M 4M 8M 16M Request size (Bytes) 14 12 10 8 6 4 SCSI 2 iscsi 0 0 512B 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M 4M 8M 16M Request size (Bytes) Fig. 6. Performance test of iscsi and (SMB) Seq output Seq input Random Per char Block Rewrite Per char Block Seek K/sec CP% K/sec CP% K/sec CP% K/sec CP% K/sec %CP K/sec %CP iscsi 12437 63 23488 8 10286 4 13002 62 22605 7 81.3 0 NFS 16015 97 19476 7 2421 1 4372 21 4800 1 123.2 0 Tab. 2. Performance test of iscsi and (NFS) A big file of 1000MB is used in Bonnie++ test. We get the similar results that iscsi provides higher performance than NFS in most cases especially block-based sequent read (Tab. 2). 3. TODAY S STORAGE ACHITECTURE AND TRADE-OFFS An ideal storage architecture would provide strong security, data sharing across platforms (i.e., operating systems), high performance, and scalability in terms of the number of devices and clients. Today s architectures force system designers to decide which of these features is most important, as choosing an architecture involves a trade-off. The three basic storage architectures in common use today are DAS, SAN and. New architectures like head and SAN file system have been introduced in an attempt to capture the features of both and SAN. DAS connects block-based storage devices directly to the I/O bus of a host machine (e.g., via SCSI or ATA/IDE). While DAS offers high performance and minimal security concerns, there are limits on connectivity. SCSI, for example, is limited by the width of the bus (a 16-bit bus can have at most 16 hosts or devices). To address the connectivity limits of DAS, and consequently enable the consolidation and sharing of storage devices, the SAN was introduced. A SAN is a switched fabric that provides a fast, scalable interconnect for large numbers of hosts and storage devices. With this added connectivity, however, came the need for better for better security. SAN therefore introduced concepts such as zoning (like a virtual private network) and host-device authentication to keep the fabric secure. DAS and SAN are both block-based. The storage application (e.g., file system) is responsible for mapping its data structures (file and directories) to blocks on the storage devices. The extra data required to do this mapping is commonly referred to as metadata. For multiple hosts to share data blocks, they must also share metadata, and do so in a manner that guarantees metadata consistency among the hosts. The complexity of this process has resulted in block sharing only among tightly coupled performance-sensitive storage applications such as clustered file systems and databases. Most other infrastructures only allow hosts to share data indirectly through files by using. Proc. of SPIE Vol. 5643 155

is just another name for file serving, which was introduced to enable data sharing across platforms. With, the metadata describing how files are stored on devices is managed completely on the file server. This level of indirection enables cross-platform data sharing but comes at the cost of directing all I/O through the single file server. may be implemented o top of a SAN or with DAS, the former often referred to as head (or gateway). In either case, clients will be limited by the performance of the file server and will rarely see the aggregate performance of the storage devices (Fig. 7b). 2 2 head SAN 1 1 SAN Metadata (a) (b) (c) Fig. 7. Symmetrical and asymmetrical architectures of /SAN combination To address the performance limitations of, SAN file systems have recently appeared. In a SAN file system, the file metadata server and clients are all connected to a SAN on which the file system is stored (Fig. 7c). Given this connectivity, the metadata server can share file metadata with the clients, thus allowing the clients to directly access the storage devices. Examples include EMC s High Road, IBM s Storage Tank, and Veritas SAN Point Direct. Because the devices have no mechanism for authorizing I/O, increasing file serving performance in this manner reduces security; the SAN mechanisms for device security only protect the entire device, not data within the device. Fig. 7a shows a general. Fig. 7b and Fig. 7c show symmetrical and asymmetrical architectures of /SAN combination respectively. The architecture illustrated in Fig. 7b is symmetrical and referred to as head. The architecture illustrated in Fig. 7c is asymmetrical and referred to as SAN file system. 1 and 1 (from metadata server s view, general servers here are also clients which require to be served) in Fig. 7c can directly access the storage devices. Through the metadata server, 2 and 2 can access the storage devices too. In the latter case, the metadata server becomes head. Each kind of architecture has its own advantages, disadvantages and applicable area. In fact, the real meaning of /SAN combination is not to use and SAN together but to combine the benefits of them. Some new emerging storage technologies such as OBS (Object Based Storage) and DAFS (Direct Access File System) have the similar considerations. The trade-off in today s architecture is therefore security and cross-platform data sharing (file) vs. high performance (blocks). While files allow one to securely share data between systems, the overhead imposed by a file server can limit performance. Yet, increasing file serving performance by allowing direct client access comes at the cost of security. Building a scalable, high performance, cross-platform, secure data sharing architecture needs to combine both the direct access nature of SAN and the data sharing and security capabilities of. 4. IP-BASED STORAGE SYSTEM FOR DATA CENTER 156 Proc. of SPIE Vol. 5643

If all storage devices are attached to a single server such as traditional file server or head, all I/O will be directed through the server. The server becomes bottleneck. As show in Fig. 8, the total throughput that users access two disks through a server is no more than users access one disk. However, if two disks depart from the server, become networkattached disks and be accessed by users directly through the network, we can think them as two. The total throughput of accessing two is approximately twice as large as accessing one disk through a server. Total Throughput (MBps) 90 80 70 60 50 40 30 20 10 0 READ with 1 Disk with 2 Disks 2 0 20 40 60 80 Total Throughput (MBps) 70 60 WRITE 50 40 30 20 10 0 0 20 40 60 80 Number of s Fig. 8. Throughput of a file server with 1 disk, with 2 disks and 2 Therefore, our basic idea to design the system is that storage devices must be accessed directly by the users without passing through a server. Fig. 9 shows the system architecture of data center. Gigabit Device iscsi Device Metadata DATA CENTER (Optional) SAN Fig. 9. System architecture of data center Proc. of SPIE Vol. 5643 157

We put hosts and storage devices in one high-speed network so that hosts can access storage devices directly. A metadata server is needed like SAN file system mentioned in part three. Data center is supposed to handle a great deal of traffic and most of them are read operations, so the asymmetrical architecture satisfies the requirements. The difference between our system and SAN file system is as follows. We use all IP-based storage technologies, and iscsi, so the storage devices can be directly connected to LAN. SAN in our system is optional. The function of the SAN is to move a large amount of data form one device to another such as LAN-free backup. For example, if a user want to read a file, the host sends request to metadata server. After metadata server sends the metadata to the host, the host read the file directly from corresponding storage devices. The and iscsi devices are plug and play and they are transparent to users. Besides providing metadata to hosts, metadata server is responsible for central management. Because we use IP-based storage technologies, the system is very cheap and relatively easy to be implemented. We use two devices and two iscsi devices to perform a test compared with traditional storage system that four devices are all attached to a single server. The results are shown in Fig. 10. The throughput of data center is more than three times as large as traditional server. Furthermore, data center is easy to be scaled. 120 100 Traditional Our Data Center Total Throughput (MBps) 80 60 40 20 0 Fig. 10. Comparison between traditional server and data center 5. CONCLUTIONS As a general communication infrastructure, IP-based technologies have experienced a widespread success that no other data communication technology has. Thus IP-based storage technologies and iscsi have great advantages. uses high-level file I/O access protocols while iscsi is a relatively low-level block I/O access protocol. iscsi can be used in both SAN and LAN. Our test results show that iscsi provides higher performance then using NFS protocol in Linux or SMB protocol in Windows ( scheme). The trade-off in today s architecture is security and cross-platform data sharing (file) vs. high performance (blocks). While files allow one to securely share data between systems, the overhead imposed by a file server can limit performance. Yet, increasing file serving performance by allowing direct client access comes at the cost of security. Building a scalable, high performance, cross-platform, secure data sharing architecture needs to combine both the direct access nature of SAN and the data sharing and security capabilities of. 158 Proc. of SPIE Vol. 5643

We design and implement an IP-based data-center-oriented storage system. The basic idea is that storage devices must be accessed directly by the users without passing through a server. From the performance test of the system, we get a good result compared with traditional server. Many problems in data center can still be further studied in the future. ACKNOWLEDGEMENTS Many people contributed to the discussions and designs of the data center system. Thanks to my colleagues for their help in conducting the experiments and the implementation of the system. REFERENCES 1. Randy H. Katz, Network-Attached Storage Systems, Proceedings of the Scalable High Performance Computing Conference-SHPCC-92, pp. 68-75, 1992. 2. Garth A. Gibson and Rodney Van Meter, Network Attached Storage Architecture, Communicatons of the ACM, vol. 43, no. 11, pp. 37-45, Nov. 2000. 3. Kaladhar Voruganti and Prasenjit Sarkar, An Analysis of Three Gigbit Networking Protocols for Storage Area Networks, Performance, Computing, and Communications, IEEE International Conference, pp. 259-265, 2001. 4. Kalman Z. Meth and Julian Satran, Features of the iscsi Protocol, IEEE Communications Magazine, vol. 41, no. 8, pp. 72-75, Aug. 2003. 5. Julian Satran, et al., iscsi (Internet SCSI), http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-20.txt. 6. UNH ISCSI Consortium, http://www.iol.unh.edu/consortiums/iscsi. 7. Mike Mesnier, Gregory R. Ganger, and Erik Riedel, Object-Based Storage, IEEE Communications Magazine, vol. 41, no. 8, pp. 84-90, Aug. 2003. 8. Shared Storage Model, http://www.snia.org/tech_activities/shared_storage_model, 2003. 9. Yingping Lu and David H. C. Du, Performance Study of iscsi-based Storage Subsystems, IEEE Communications Magazine, vol. 41, no. 8, pp. 76-82, Aug. 2003. 10. Stephen Aiken, Dirk Grunwald, and Andrew R. Pleszkum, A Performance Analysis of the iscsi Protocol, Proceedings of the 20 th IEEE/11 th A Goddard Conference on Mass Storage Systems and Technologies (MSS 03), pp. 123-134, 2003. 11. Jerry Sievert, Iometer: The I/O Performance Analysis Tool for s, http://www.intel.com/design/servers/devtools/iometer/index.htm. 12. Tim Bray, textuality - Bonnie, http://www.textuality.com/bonnie/. 13. Huseyin Simitci, Storage Network Performance Analysis, Wiley Publishing, Inc., 2003 14. Abraham Silberschatz, Peter Baer Galvin, and Greg Gagne, Operating System Concepts, Sixth Edition, Wiley Text Books, 2002. Proc. of SPIE Vol. 5643 159