東 海 大 學 資 訊 工 程 研 究 所 碩 士 論 文

Transcription

1 東 海 大 學 資 訊 工 程 研 究 所 碩 士 論 文 指 導 教 授 : 楊 朝 棟 博 士 以 異 質 儲 存 技 術 實 作 一 個 軟 體 定 義 儲 存 服 務 Implementation of a Software-Defined Storage Service with Heterogeneous Storage Technologies 研 究 生 : 連 威 翔 中 華 民 國 一 零 三 年 七 月

2 1

3 摘 要 雲 端 技 術 的 應 用 和 服 務 與 日 俱 增, 無 論 是 政 府 企 業 或 是 個 人 都 可 能 有 建 構 雲 端 系 統 的 需 求 隨 著 雲 端 運 算 的 盛 行, 雲 端 基 礎 設 施 的 需 求 也 跟 著 提 升, 要 如 何 以 最 少 的 成 本 達 到 最 佳 的 效 果, 是 眾 人 所 面 臨 的 一 大 問 題, 而 發 展 各 種 的 虛 擬 化 技 術 便 是 其 中 的 一 種 解 決 方 法, 將 資 源 做 邏 輯 上 的 分 割 轉 變 為 邏 輯 上 可 以 管 理 的 資 源, 並 提 供 給 需 求 者, 讓 使 用 者 更 合 理 更 充 分 的 控 制 與 管 理 這 些 資 源 軟 體 定 義 儲 存 (Software-Defined Storage, SDS) 便 是 虛 擬 化 技 術 中 的 一 項, 透 過 軟 體 將 儲 存 資 源 集 中 化, 整 合 異 質 儲 存 設 備, 提 升 現 有 儲 存 設 備 的 可 用 度 與 使 用 率, 並 且 可 以 視 使 用 者 需 求 調 整, 以 達 到 最 佳 的 效 能 近 年 來 SDS 的 概 念 逐 漸 被 大 家 所 注 意, 各 家 廠 商 也 紛 紛 推 出 自 己 的 產 品, 但 始 終 沒 有 一 個 公 認 的 標 準 出 現, 多 數 方 案 仍 是 偏 向 於 自 家 的 商 品,SDS 能 夠 整 合 的 儲 存 只 限 於 小 範 圍 本 論 文 將 採 用 以 OpenStack 作 為 架 設 雲 端 的 環 境, 並 在 OpenStack 上 整 合 HDFS Ceph Swift 等 儲 存, 以 達 到 軟 體 定 義 儲 存 的 概 念 本 論 文 採 用 的 軟 體, 將 可 以 整 合 不 同 儲 存 設 備, 提 供 一 個 整 合 性 的 儲 存 陣 列, 建 立 虛 擬 儲 存 池, 使 用 戶 感 覺 不 到 儲 存 硬 體 的 限 制 並 且 提 供 管 理 者 一 個 網 頁 介 面, 將 可 讓 管 理 者 在 不 關 閉 機 器 的 情 況 下 調 整 儲 存 空 間 及 管 理 用 戶 或 安 全 設 定 等 等 而 效 能 測 試 方 面 最 後 以 Hadoop HDFS 效 能 最 佳, 除 了 少 數 部 分, 整 體 而 言 是 優 於 其 他 兩 者, 是 較 佳 使 用 於 生 產 環 境 中 的 儲 存 關 鍵 字 : 雲 端 服 務, 虛 擬 化, 軟 體 定 義 儲 存 I

4 Abstract With the popularity of cloud computing, the demand of cloud infrastructure also increases. Because the requirement of cloud computing increases steeply, the infrastructure of cloud service expands too. Spending the least cost to achieve the best effect on cloud is a big problem for every researcher, and virtualization is one of the answers to resolve this problem. Virtualization is a technology that divides the real resources into logically manageable resources, which are then provided to users to enable the users control and manage them easily and legitimately. Software-Defined Storage (SDS) is a kind of virtualization, in which by integrating the storage resource and different storage devices by software to increase the usability and activity, the user can according to their requirement to make adjustment and then achieve the optimal performance. In recent years, SDS becomes more and more popular, and several companies have announced their product. But the generic standard still has not appeared; most products are only appropriate for their devices and SDS just can integrate a few storages. In this thesis, we will use the OpenStack to build and manage the cloud service, and use software to integrate storage resources include Hadoop HDFS, Ceph and Swift on OpenStack to achieve the concept of SDS. The software used can integrate different storage devices to provide an integrated storage array and build a virtual storage pool; so that users do not feel restrained by the storage devices. Our software platform also provides a web interface for managers to arrange the storage space, administrate users and security settings. For allocation of the storage resources, we make a policy and assign the specific storage array to the machine that acquires the resource according to the policy. Then we run performance tests of file systems to prove II

5 our system will run correctly. Finally, according to the experimental results, we find the performance of Hadoop HDFS is better than the other two storages; i.e., except a few instances, Hadoop HDFS out performs the other two storages in the same environment. Keywords: Cloud service, Virtualization, Software-Defined Storage III

6 致 謝 詞 能 夠 有 機 會 並 且 順 利 的 完 成 這 篇 論 文, 並 非 我 獨 自 一 人 能 夠 做 到 的, 其 中 還 包 含 了 許 許 多 多 來 自 各 方 面 的 支 持 協 助, 若 非 有 這 些 無 論 是 物 質 亦 或 者 是 精 神 上 的 幫 助, 我 沒 有 辦 法 達 成 這 樣 的 目 標 首 先 我 要 感 謝 我 的 指 導 教 授 楊 朝 棟 老 師, 謝 謝 老 師 六 年 來 的 教 導, 引 領 我 進 入 現 在 的 研 究 領 域, 老 師 以 其 言 傳 身 教, 教 導 我 研 究 的 精 神 和 做 事 的 態 度 以 及 方 法, 在 學 習 的 過 程 中 督 促 並 指 點 我 正 確 的 方 向, 讓 我 在 一 件 件 的 工 作 中 學 習 與 成 長, 若 沒 有 這 些 磨 練, 我 便 沒 有 辦 法 順 利 完 成 這 篇 論 文 與 學 業 也 希 望 在 最 後 的 成 果 上, 沒 有 讓 老 師 失 望 感 謝 各 位 口 試 委 員 在 口 試 的 過 程 中, 指 導 協 助 我 改 正 並 完 善 我 的 論 文, 感 謝 薛 念 林 老 師 朱 正 忠 老 師 盧 志 偉 老 師 張 志 宏 老 師, 不 辭 辛 勞 撥 空 擔 任 我 的 口 試 委 員, 提 供 給 了 我 許 多 的 建 議, 因 為 有 你 們 的 幫 助, 令 我 的 論 文 能 夠 更 加 的 完 善 感 謝 實 驗 室 的 學 長 們, 在 剛 進 入 實 驗 室 時, 以 身 作 則 引 領 我 適 應 研 究 生 的 生 活, 感 謝 各 位 同 學 們 在 兩 年 來 在 諸 多 方 面 的 幫 助, 無 論 是 在 技 術 方 面 討 論 論 文 方 向 計 劃 的 執 行 等 等, 因 為 有 你 們 在, 讓 我 可 以 順 利 的 度 過 研 究 所 的 生 活 感 謝 學 弟 妹 們 的 幫 助, 在 這 最 後 一 段 時 間, 幫 忙 分 攤 實 驗 室 的 工 作, 令 我 能 夠 專 注 在 論 文 上 感 謝 父 母 對 我 從 小 的 栽 培, 因 為 有 你 們 一 路 上 的 支 持, 讓 我 順 利 的 度 過 我 的 求 學 生 活, 讓 我 有 這 個 機 會 能 夠 站 在 這 裡, 作 我 想 要 的 研 究 最 後, 僅 以 此 文 獻 給 所 有 一 路 上 幫 助 我 陪 伴 我 的 人, 因 為 有 你 們 才 有 今 天 的 我, 謝 謝 你 們 IV

7 Table of Contents 摘 要 Abstract 致 謝 詞 Table of Contents List of Figures List of Tables I II IV V VII VIII 1 Introduction Motivation Thesis Goal and Contributions Thesis Organization Background Review and Related Work Cloud Computing Virtualization Full Virtualization Para-virtualization Hardware-assisted Virtualization Storage Virtualization Software-Defined Storage OpenStack Swift HDFS Ceph Related Work System Design and Implementation System Architecture System Implementation Experimental Results 29 V

8 4.1 Experimental Environment Performance of Data Transport User Interface of ViPR Discussion Conclusions and Future Work Conclusion Remarks Future Work References 43 A Hadoop Installation 48 B Ceph Installation 52 C GlusterFS Installation 55 VI

9 List of Figures 2.1 Full Virtualization Para-virtualization Hardware-assisted Virtualization the storage situation compare from the before and now OpenStack The HDFS architecture System Architecture Ceph Architecture HDFS Service Architecture HDFS Architecture OpenStack Overview VM Instances ViPR with OpenStack Architecture ViPR System Architecture ViPR Cluster Write single file with different file size Read single file with different file size Write small file Read small file Write small file Read small file Dashboard Add PhysicalAsset Add Host Add VirtualAsset VirtualPool Create DataService Setup VII

10 List of Tables 4.1 Hardware Specification Software Specification VIII

11 Chapter 1 Introduction Cloud computing, one of today s hottest topics, is being developed very fast. There are many large companies such as Google, Amazon, and Yahoo offering cloud services to millions of users at the same time. A lot of people including programmers in the enterprise, government, and research centers, or even some common people, want to build cloud computing environments. But the operating cost of a cloud may be very expensive; the cost includes acquirement and maintenance of hardware facilities, and salaries paid to operating engineers and so on. So how to reduce the cost is a big issue. 1.1 Motivation Cloud computing is a concept, in which computers over networks are able to cooperate with each other to provide far-reaching network services [7, 8, 19, 32, 36]. The basic approach of cloud computing is to computing through terminal operations over the Internet by moving hardware, software, and shared information from users to the server side. Due to this situation, we can say that we are living in a world that is surrounded by the cloud technology, and we almost use the cloud every day. Now, more and more enterprises join the procession of cloud service providers. There are many kinds of cloud services in the market. When a 1

12 company wants to build a cloud, a difficult problem it must face is how to choose the right cloud product to suit its need. Software-Defined Storage (SDS) is a kind of virtualization which integrates the storage resource and different storage device by software to increase the usability and activity; the users can adjust SDS according to their requirement to achieve the optimal performance. In recent years, SDS is an increasingly popular concept, and several companies have announced related products. But the generic standard still has not formed; hence, most of its products are only appropriate for some devices. 1.2 Thesis Goal and Contributions In this thesis we implement a cloud service to integrate several heterogeneous storage technologies. In order to achieve the goal of high availability, we choose the technologies with a redundant mechanism, in which crash of machines is considered a normal situation and mechanisms exist to resolve it. It will provide high availability to this system. In addition to this, we use the open source technology to let the system have good compatibility. It can also help the users to freely build their systems with high flexibility. These features are present in the design concept of Software-Defined Storage. 1.3 Thesis Organization Chapter 2 will describe some background information and related works, including cloud computing, virtualization, software-defined storage. Chapter 3 will introduce our experimental environment and methods, and the overall architecture. Chapter 4 will list the hardware information measurement method and software version, present and analyze experimental results. Finally, Chapter 5 summarizes this thesis by pointing out its major contributions and future work directions. 2

13 Chapter 2 Background Review and Related Work 2.1 Cloud Computing Cloud computing is Internet-based computing, in which shared hardware, software resources, and data can be provided to users according to their demand of computers and other devices. The user does not need to know the details of the cloud infrastructure, or has appropriate expertise, and is without direct control. Cloud computing allows companies to fast deploy applications and reduce the complexity of management and maintenance costs, and enables rapid changes of IT resources reallocation to respond business needs. Cloud computing, usually involving with the Internet, describes a new Internet-based services to increase IT use and delivery models, and is easy to provide dynamic and virtual extension of the resource. The cloud is a metaphor of networks or the Internet. Cloud computing can be considered to include the following levels of service: infrastructure as a service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS): SaaS: Consumers use the application, but do not control the operating system, hardware, or the operation of the network infrastructure, for example: 3

14 the Microsoft CRM or Salesforce.com platform as a service. With the SaaS model, the user can access the services, software, and data. The service providers maintain the normal operation of the infrastructure and platform to maintain service, while users access cloud services through browsers, and desktop or mobile applications. PaaS: Consumers use the host operating applications. Consumers control the operation of the application environment (host also has some control over it), but do not control the operating system, hardware, or the operation of the network infrastructure. The platform is usually the application infrastructure, for example: the Google App Engine. IaaS: Consumers can use the basic computing resources such as processing power, storage space, network components or middleware. Consumers can control the operating system, storage, deployed applications and network components (e.g. firewalls, load balancers), but do not control the cloud infrastructure. Examples of IaaS include Amazon AWS, and Rackspace. Cloud computing relies on the sharing of resources in order to achieve economies of scale in similar infrastructure. Service providers integrate a large number of resources for use by multiple users. Users can easily request more resources and adjust usage at any time, without the need to release all resources back to the whole structure. Users do not need to buy a lot of computing resources; they only need to enhance the amount of rent for short-term spikes of resource demands, and release some resources during low demand periods. Service providers are able to lease unused resources to other users, and even adjust rents in accordance with the demand of the whole. The National Institute of Standards and Technology (NIST) also classifies cloud computing based on the deployment model. Public cloud: In short, public cloud provides services through the Internet and third party service provider, and is opening up to the use of customers. The term public does not necessarily mean free, but may represent very cheap. In the public cloud, the user data is available for anyone to view. 4

15 Public cloud providers usually suggest interested users to implement some access control mechanisms. The public cloud is a flexible and cost-effective solution. Private cloud: Private cloud have many advantages, such as with more flexibility than a public cloud environment and being suitable for the provision of services. The difference between the two is that for private cloud services, data and programs are managed within the organization, while for public cloud services, they are constrained by the network bandwidth, security concerns, and regulatory restrictions. In addition, private cloud services suppliers and users have more control over the cloud infrastructure to improve safety and flexibility, because the user and the Internet are subject to special restrictions. Community Cloud: Community cloud is controlled and used by members of organizations with common interests, particular security requirements, and common purpose. Community members can access cloud data and applications. Hybrid cloud: The hybrid cloud is a mix of the public cloud and private cloud. In this mode, the user usually has non-business-critical information dealt with in a public cloud, but at the same time, control of business-critical services and information. 2.2 Virtualization With virtualization, the computer s physical resources, such as servers, network, memory, and storage, are abstracted after conversion, so that users can apply those resources in a better way than the original configuration. Virtualization is commonly referred to virtualized resources including computing power and data storage; in this thesis virtualization is specifically referred to virtualization of servers [12, 15, 24, 30, 33]. The server virtualization software technology refers to 5

16 the use of one or more of the host hardware settings. It has the flexibility to configure the virtual hardware platform and operating system, like real hardware. In this way, a variety of different operating environments (for example, Windows, Linux, etc.) can operate simultaneously on the same physical host, and be independently operating in different physical hosts. Virtualization solutions can be broadly divided into three categories: full virtualization, para-virtualization, and hardware-assisted virtualization Full Virtualization In full virtualization, Hypervisor replaces the traditional core of the operating system in the Ring 0 privilege level, via Binary Translation technology; the guest operating system requires direct access to the hardware Ring 0 level instruction by the hypervisor conversion to submit requests further to hardware. Hypervisor virtualizes all hardware components. The guest operating system works as the individual real host, with a high degree of independence. But Binary Translation technology will cause the performance of the virtual machine (VM) to decrease significantly. One example of software using virtualization technology is the VMware Workstation. The full virtualization is totally dependent on the virtual hardware layer created Guest OS, which can use hardware resources, only limited by the virtual hardware resources, and less able to misallocate of the entity s computer hardware. In addition to the CPU, memory, the graphics card, etc. can also be virtualized if the proprietary driver has already been installed. The benefits of full virtualization are, regardless of how the hardware environment, the Guest OS is able to maintain a more consistent compatibility, and can use different operating systems with the physical machine. The disadvantage of full virtualization is that it is a greater burden of the physical machine. 6

17 Figure 2.1: Full Virtualization Para-virtualization Para-virtualization, also known as parallel virtualization, is not all hardware device virtualization such as Xen. Compared to the full virtualization virtual machine instruction Binary Translation, the para-virtualized VM invokes hardware devices through Dom0, and manages each VM access to physical resources. The VM performance is significantly improved, but the hardware driver binding in dom0, and the core of the operating system in the VM must go through a special modification; thus the independence of the VM is relatively low. The para-virtualization does not configure the virtual layer on top of the hardware, but use more than one memory location program calls at different times. Para-virtualization allows the Guest OS share hardware resources. The advantage is that the hardware will not need to waste performance on the hardware layer; however, the drawback is that the operating system in the virtual machine must be consistent with the physical environment. 7

18 Figure 2.2: Para-virtualization Hardware-assisted Virtualization Hardware-assisted virtualization, as shown in Figure 2.3, is used to overcome the dilemma that the virtual machine operating system kernel cannot be placed in the processor Ring 0 privilege level. Since both the hypervisor and virtual operating system kernel can be issued in Ring 0, the hypervisor will be automatically intercepted by the need to deal with the instruction of the virtual operating system directly by the hardware processing; therefore the full virtualization binary translation or para-virtualization Hypercall operations are no longer in need. Hardware assisted virtualization basically eliminates the difference between full virtualization and para-virtualization. On the basis of hardware-assisted full virtualization or para-virtualization program virtual machines are high performance and independent. The representative program of the hardware-assisted virtualization VMware is ESXi with open source Kernel-based Virtual Machine (KVM). KVM needs a host operating system, whose core provides virtualization services; whereas VMware ESXi is the equivalent of a specialized virtualization thin client operating system, installed directly on a physical machine. 8

19 Figure 2.3: Hardware-assisted Virtualization Storage Virtualization After the server virtualization, the interactive mode of the server and storage device changed from one to one to the same application with several VMs to one storage device. It causes read/write of data more complicated like Figure 2.4. In addition to the application which has to request immediately such as DB or ERP, there are more and more new type of cloud and big data applications to be released like Node.js, Phthon and Hadoop and so on. Because the appearance of these new type technology, the ability of storage device to support them is challenged time and again. Additionally, there are several kinds of different architecture and medium of storage. For instance the new architecture like NAS and SAN or the medium like SATA, SAS, SSD and many other storage architecture developed for virtualization; the more and more new technologies let situations of data services become more complicated [28, 29]. 9

20 Figure 2.4: the storage situation compare from the before and now Software-Defined Storage Software-Defined Storage (SDS) is a term for the computer data storage technology which separates storage hardware from the software that manages the storage infrastructure. The software enabling SDS environments can provide policy management for feature options such as deduplication, replication, thin provisioning, snapshots and backup [3, 23, 37]. Characteristics of SDS could include any or all of the following features [6,22]: Abstraction of logical storage services and capabilities from the underlying physical storage systems, and in some cases, pooling across multiple different implementations. Since data movement is relatively expensive and slow compared to compute and services (the data gravity problem in infonomics), pooling approaches sometimes suggest leaving it in place and creating a mapping layer to it that spans arrays. Examples include Automation with policy-driven storage provisioning with service-level agreements replacing technology details. This requires management interfaces 10

21 that span traditional storage array products, as a particular definition of separating the control plane from data plane, in the spirit of OpenFlow. Prior industry standards efforts include the Storage Management Initiative Specification (SMI-S) which began in Commodity hardware with storage logic abstracted into a software layer. This is also described as a clustered file system for converged storage. Scale-out storage architecture. 2.3 OpenStack OpenStack [20] is an IaaS cloud computing project for public and private clouds. It is free open source software released under the terms of the Apache License. The project aims to deliver solutions for all types of clouds by being simple to implement, massively scalable, and features rich. The technology consists of a series of interrelated projects delivering various components for a cloud infrastructure solution. Founded by Rackspace Hosting and NASA, OpenStack has grown to be a global software community of developers collaborating on a standard and massively scalable open source cloud operating system. Its mission is to enable any organization to create and offer cloud computing services running on standard hardware. The project is managed by the OpenStack Foundation, a non-profit corporate entity established in September 2012 to promote, protect and empower OpenStack software and its community. OpenStack offers flexibility and choice through a highly engaged community of over 6,000 individuals and over 190 companies including Rackspace, such as Intel, AMD, Canonical, SUSE Linux, Inktank, Red Hat, Groupe Bull, Cisco, Dell, HP, IBM, NEC, VMware and Yahoo. It is portable software, but is mostly developed and used on operating systems running Linux. The technology consists of a series of interrelated projects that control large pools of processing, storage, and networking resources through-out a datacenter, 11

22 all managed through a dashboard that gives administrators control while empowering its users to provision resources through a web interface. OpenStack is committed to an open design and development process. The community operates around a six-month, time-based release cycle with frequent development milestones. During the planning phase of each release, the community gathers for the OpenStack Design Summit to facilitate live developer working sessions and assemble the roadmap. Figure 2.5: OpenStack 2.4 Swift OpenStack Object Storage (Swift) is a scalable redundant storage system. Objects and files are written to multiple disk drives spread throughout servers in the data center, with the OpenStack software responsible for ensuring data replication and integrity across the cluster. Storage clusters scale horizontally simply by adding new servers. Should a server or hard drive fail, OpenStack replicates its content from other active nodes to new locations in the cluster. 12

23 Because OpenStack uses software logic to ensure data replication and distribution across different devices, inexpensive commodity hard drives and servers can be used. 2.5 HDFS The Hadoop Distributed File System (HDFS) is an Apache Software Foundation project and a subproject of the Apache Hadoop project. HDFS is a distributed file system designed to hold very large amounts of data (terabytes or even petabytes), and to provide high-throughput access to this information. Files are stored in a redundant fashion across multiple machines to ensure durability to failure and high availability to parallel applications [39]. HDFS has many similarities with other distributed file systems, but is different in several respects. One noticeable difference is HDFS s write-once-read-many model that relaxes concurrency control requirements, simplifies data coherency, and enables high-throughput access [4, 16, 26, 38]. Another unique attribute of HDFS is the viewpoint that it is usually better to locate processing logic near the data rather than moving the data to the application space. HDFS rigorously restricts data write to one write at a time. Bytes are always appended to the end of a stream, and byte streams are guaranteed to be stored in the written order. HDFS is comprised of interconnected clusters of nodes where files and directories reside. An HDFS cluster consists of a single node, known as a NameNode, which manages the file system namespace and regulates client access to files. In addition, data nodes (DataNodes) store data as blocks within files. Within HDFS, a given name node manages file system namespace operations like opening, closing, and renaming files and directories. A name node also maps data blocks to data nodes that handle read and write requests from HDFS clients. Data nodes also create, delete, and replicate data blocks according to instructions from the governing name node. The architecture of HDFS is shown in Figure

24 Figure 2.6: The HDFS architecture 2.6 Ceph Ceph [2, 34, 40] is a software storage platform designed to present object, block, and file storage from a single distributed computer cluster. Ceph is a distributed storage designed to provide excellent performance, reliability and scalability. Ceph was made possible by a global community of enthusiastic storage engineers and researchers. It is open source and freely-available. Ceph software runs on commodity hardware. The system is designed to be both self-healing and self-managing and strives to cut both administrator and budget costs. Object Storage: Ceph is a distributed object storage and file system designed to provide excellent performance, reliability and scalability. Its software libraries offer client applications with direct access to the reliable autonomic 14

25 distributed object store (RADOS) object-based storage system, and also provide a basis for some of Ceph s advanced features, including RADOS Block Device (RBD), RADOS Gateway, and the Ceph File System. The librados software libraries enable applications written in C, C++, Java, Python and PHP. The RADOS Gateway also exposes the object store as a RESTful interface which can present as both native Amazon S3 and Open- Stack Swift APIs. The librados libraries provide advanced features, including: Partial or complete reads and writes Snapshots Atomic transactions with features like append, truncate and clone range Object level key-value mappings Block Storage: Ceph s object storage system allows users to mount Ceph as a thinly provisioned block device. Ceph s RADOS Block Device (RBD) provides access to block device images that are striped and replicated across the entire storage cluster. When an application writes data to Ceph using a block device, Ceph automatically stripes and replicates the data across the cluster. Ceph s RADOS Block Device (RBD) also integrates with KVMs, bringing Ceph s virtually unconstrained storage to KVMs running on user s Ceph clients. Ceph RBD interfaces with the same Ceph object storage system that provides the librados interface and the CephFS file system, and it stores block device images as objects. Since RBD is built on top of librados, RBD inherits librados s capabilities, including read-only snapshots and revert to snapshot. Ceph s object storage system is not bounded to native binding or RESTful APIs. User can mount Ceph as a thinly provisioned block device. When write data to Ceph using a block device, Ceph automatically stripes and replicates the data across the cluster. By striping images across the cluster, Ceph increases read access performance for large block device images. 15

26 File System: Ceph s file system (CephFS) runs on top of the same object storage system that provides object storage and block device interfaces. Ceph provides a POSIX-compliant network file system that aims for high performance, large data storage, and maximum compatibility with legacy applications. Compared to many object storage systems available today Ceph s object storage system offers a significant feature: a traditional file system interface with POSIX semantics. Object storage systems are a significant innovation, but they supplement rather than replace traditional file systems. The Ceph metadata server cluster provides a service that maps the directories and file names of the file system to objects stored within RADOS clusters. The metadata server cluster can expand, contract, and dynamically rebalance the file system to distribute data evenly among cluster hosts. As storage requirements grow for legacy applications, organizations can configure their legacy applications to use the Ceph file system. This means user can run one storage cluster for object, block and file-based data storage. This ensures high performance and prevents heavy loads on specific hosts within the cluster. 2.7 Related Work Chengzhang Penga and Zejun Jiangb [21] proposed a cloud storage service system, in which a solution is suggested to build a cloud storage service system based on the open-source distributed database. Their system was designed as three-tiered architecture. And they demonstrated their prototype of CS3 built based on the framework. A file system architecture for organization of data and metadata that will efficiently organize and enable sharing besides exploiting the power of storage virtualization and maintaining simplicity in such a highly complex and virtualized environment proposed by Ankur Agrrawal et al [1]. 16

27 Tahani Hussain et al. [10] assessed the performance of an existing enterprise network before and after deploying distributed storage systems. And the simulation of an enterprise network with 680 clients and 54 servers followed by redesigning shows improvements in the storage system throughput by 13.9% with 24.4% decreased average response time and 38.3% reduced packet loss rate. Dejun Wang [31] proposed an efficient cloud storage mode for heterogeneous cloud infrastructures. And he verified by extensive tests to the model with numerical examples. He thought cloud storage system with traditional storage systems has some differences; such as the demand from the performance point of view, data security, reliability, efficiency and other indicators need to be considered for cloud storage services, which are services in a wide range of complex network environment for the demands of large-scale users. 17

28 Chapter 3 System Design and Implementation 3.1 System Architecture The main goal of this thesis is to build a cloud service platform integrating with several heterogeneous storage technologies. The cloud storage combines different storage technologies that are the most popular and perform great in the same type of technology. These storages that we chose are open source; hence, it will be helpful for further development and maintenance in the future. Additionally, we will offer a simple GUI for users to manage the cloud service. In this chapter, we will overview the whole system. 18

29 Figure 3.1: System Architecture The system architecture consists of some components. The first part is a server node built by OpenStack. And we also used OpenStack to build some VMs to be the storage nodes. Second, we used three kinds of different storage technology to build the storage of OpenStack. And we used software ViPR to control the different storage devices and manage storage spaces for the storage of OpenStack. ViPR will display the storage with a simple appearance and it can be used to integrate different storage technologies. The system architecture is shown in Figure 3.1. In our experiment, we focus on the inside of system. The basic of system was built by OpenStack, which was combined with several components, including the computing side, network side and storage side. We integrated some different 19

30 storage to be the storage side for the OpenStack cloud. And the storage controller integrated them by the APIs of these storage. 3.2 System Implementation Architecture of Ceph: there are three daemons in the Ceph side, including OSD, MDS and MON. The OSD is responsible for saving data as an object; it used an independent partition, and it will partition to a xfs type. It at least has one node, and the capacity of Ceph is defined by the Equation(3.1), where i means the number of osd from 0 to n, and n means the last number of OSD. The function shows the total capacity of OSD as the sum of all OSDs. CephSize = n partitionsize(osd) i, n 1 (3.1) i=0 The second is MDS, responsible for storing the data structure of the data index, and writing it into the OSD. It at least has one node, and it uses a redundant mechanism. The last is MON which is responsible to monitor the running status of OSD and MDS, and according to the situation to adjust them. It at least has one node, and it uses a redundant mechanism too. The Ceph architecture is shown in Figure

31 Figure 3.2: Ceph Architecture Every OSD has an independent partition; and the three daemons will not conflict with one another. They can run on a same node. Architecture of HDFS: the HDFS is combined with the master node and the slave node which is called NameNode and DataNode. And we built a HDFS architecture consisting of one NameNode and two DataNodes shown in Figure 3.3. The NameNode executes file system namespace operations like opening, closing, and renaming files and directories. It also determines the mapping of blocks to DataNodes. The DataNodes are responsible for serving read and write requests from clients of the file system. The DataNodes also performs block creation, deletion, and replication upon instruction from the NameNode shown in Figure

32 Figure 3.3: HDFS Service Architecture Figure 3.4: HDFS Architecture 22

33 OpenStack: OpenStack was used to build our cloud; it is open source software to manager cloud. It was built on a VM with Ubuntu OS, and then some VMs were created to form a storage system cluster. The overview of the system is shown in Figure 3.5 and Figure 3.6. Figure 3.5: OpenStack Overview Figure 3.6: VM Instances EMC ViPR: traditional data center is just a set of some servers, storage, network and security and etc. In the traditional data center, to build a new application usually takes several weeks. But the process can be simplified by the data center driven by completely dynamic software, making it like building a new VM. If we can design a series of policies that can be 23

34 implemented on a variety of new applications to automatically support infrastructure services included in a container, i.e., a virtual data center, then, in a few minutes or a few seconds, we can complete the deployment of a new application. EMC Virtualization Platform Reinvented (ViPR) is a logical storage system, not a physical storage. It can integrate EMC storage and third-party storage in a storage pool, and manage it as a single system, but still save the worth of original storage. ViPR can replicate data across several different place and data center with different store product, and it provides a unified block store, object store and file system and other services. At the same time, ViPR provides a unified metadata service and self-service deployment, measurement and monitoring services. In addition to this, ViPR also applies to multi-tenant environments. EMC proposed a model about software-defined storage, and described the relation with OpenStack. The architecture is shown in Figure 3.7. Figure 3.7: ViPR with OpenStack Architecture 24

35 ViPR run on three to five virtual server machines. It consists of the control plane and data plane; the first part realizes automatic store, and the second part provides data service by building on the first part. ViPR provides some APIs to let users to use services and manage storage, as shown in Figure 3.8. One part of ViPR is the control plane. It can manage all applications of storage array, such as data mining, storage resource searching, and capacity counting and reporting. The controller controls the application of virtual storage array, and manages the storage resource. In order to achieve the goal, ViPR virtualizes the control path of physical storage, and runs the function with it. By the virtual plane, people can manage the storage pool and split it to different virtual storage arrays with policies, just like virtualization of the server. In addition to the controller and data service, ViPR provides the open RESTful API. The developers can develop new services without the limit of the hardware. Through these APIs, people can access data, and add, delete, modify, monitor and measure the logical storage resources. 25

36 Figure 3.8: ViPR System Architecture ViPR is built by the scale-out architecture; if it is built with at least three clusters, the architecture will provide the ability of high availability, load balance and system upgrade and so on. It provides RESTfulAPI, GUI(console), CLI and SDK to let user control it with high flexibility as shown in Figure 3.9. And ViPR can provide the automatic of disaster recovery. Through the continuous remote replication technology, it can provide the ability of data replication continuously to successfully achieve the goal of disaster recovery. 26

37 Figure 3.9: ViPR Cluster The data service of ViPR has some features: Unified Storage: ViPR offers the traditional layer on top of cloud storage data type, with the following advantages: tradition by allowing local access to the underlying storage, improved compatibility with existing applications; while allowing manipulation of data at the right time through different methods to improve the efficiency of the work; and it can convert existing data to the cloud easily. Heterogeneous Storage: ViPR provides an engine for a variety of storage devices, for a given environment or use case can select the appropriate storage capacity and hardware, and the user can reuse the existing storage hardware investments, but also applies to mixed ViPR additional scenarios, such as across different devices tiered storage and replication. 27

38 Enterprise-grade Storage: cloud storage platforms often lack some of the features needed in enterprise scenarios, such as snapshots and compliance. The ability to leverage and extend ViPR data services to the underlying storage devices to provide enterprise-class cloud storage, such as ViPR can provide incremental real-time snapshot of the object store. Flexible Storage: ViPR data services are implemented by software, which makes ViPR data service can be deployed on any server in the data center, in simple, reliable, lightweight and scalable design; And while all of these functions underlying storage devices, IP is not locked in any storage array. Extensible Storage: ViPR data services provides a rich API, third-party services can use it to develop their own systems. In addition, ViPR will also expose the raw building blocks, these modules can provide the core IP cloudscale services, including distributed B + tree, metadata; third party can use these to develop services that modules can be placed inside or outside ViPR to extend a platform. Object Storage: object storage services like Amazon S3 ViPR provided the model for the image. ViPR APIs support AmazonS3, OpenStackSwift and EMC s Atmos, vnextapi support EMCCentera CAS. The ViPR also provides some extensions, including Byte range updates, atomic increase, rich ACL snapshots, metering and billing, and so on. In addition, ViPR provides such functions as file can be accessed directly on the same underlying file storage device object storage, and the performance of the local file system consistency. HDFS: the HDFS file system has become increasingly popular, not only in Hadoop, but also in a variety of distributed applications. ViPR HDFS provides massive storage platform; ViPR achieves through existing HDFS to solve the problem addressing key storage engine, in addition ViPR design can easily be stored in HDFS into existing hardware environment. One can also make HDFS run target storage of above scenarios. 28

39 Chapter 4 Experimental Results 4.1 Experimental Environment We used OpenStack to build our cloud platform, which then was used to create and manage the distributed storage system. In the system we integrated three heterogeneous storage technologies. And we built the storage system by some VMs, in which Ceph was constructed by three VMs with specifications of 1 core CPU, 2GB memory, and a total of 60GB storage space. The first node was MON and OSD, the second was MDS and OSD, the third was OSD. These were components in the Ceph cluster. And the HDFS part was constructed by three virtual machines including one NameNode and two DataNodes with specifications of 1 core CPU, 2GB memory, and total of 40GB storage space. 4.2 Performance of Data Transport In the first part of our experiment, we compared the three different kinds of storage technology, including Ceph, Hadoop HDFS and GlusterFS. The reason to choose them is that they are famous storage technology in the popular distributed storage systems today. 29

40 Table 4.1: Hardware Specification Host name CPU Memory Disk OS Management node vcpu 9 cores 20GB 200GB Ubuntu Ceph node1 1 cores vcpu 2GB 20GB Ubuntu Ceph node2 1 cores vcpu 2GB 20GB Ubuntu Ceph node3 1 cores vcpu 2GB 20GB Ubuntu HDFS Namenode1 1 cores vcpu 2GB 20GB Ubuntu HDFS Datanode1 1 cores vcpu 2GB 20GB Ubuntu HDFS Datanode2 1 cores vcpu 2GB 20GB Ubuntu GlusterFS server1 1 cores vcpu 2GB 20GB Ubuntu GlusterFS server2 1 cores vcpu 2GB 20GB Ubuntu GlusterFS client 1 cores vcpu 2GB 20GB Ubuntu Table 4.2: Software Specification Software Version OpenStack Havana Ceph V0.72 EMPEROR HDFS Swift GlusterFS 3.5 In this experiment, we built three storage systems: Hadoop, Ceph and GlusterFS and recorded the data of the performance of uploading and downloading files on the three systems. We used three cases to upload and download a single file. The first case was for files with the size of 1MB to 16GB. The second is tests of small sized files of 1MB to 100MB, with 16GB total file size. And the last one is files100mb to 1000MB, with 16GB total file size. The result for the experiment to upload a single file from 1MB to 16GB from the client node to the server node is shown in Figure 4.1. We observe that when the file size is smaller than 512MB, the performance of the three storage systems are 30

41 similar. But when the file size is larger than 512MB, the performance of Hadoop is better than GlusterFS and Ceph. According to this result, we can say the ability of Hadoop is better than GlusterFS and Ceph when uploading large files. Figure 4.1: Write single file with different file size The result for the experiment to download a single file from 1MB to 16GB from the client node to the server node is shown in Figure 4.2. From the result, we observe that when the file is smaller than 512MB, the performance of the three storage systems are similar. But when the file size is larger than 512MB, the performance of GlusterFS is better than the other two. However, when the file size is larger than 4GB, the performance of Hadoop is better than Ceph. 31

42 Figure 4.2: Read single file with different file size In this part, we uploaded 16GB file in total, and split the file into smaller files of 10MB to 100MB. We will use it to test the effect of performance when transporting files of different sizes. In Figure 4.5, we can see the curve is almost stable; however when the file size is just larger than 60MB, the run time of Hadoop increase fast. And when the file size is between 30MB to 60MB, its performance is the best. The performance of GlusterFS and Ceph are similar to each other. And Hadoop performs better than they. 32

43 Figure 4.3: Write small file In this part, we downloaded 16GB file in total, and split the file to be 10MB to 100MB. We will use it to test the effect of performance of downloading different file sizes, as shown in Figure 4.6. In this experiment, we find the results have not consistent trends; they are close. The performance of Hadoop is better than Ceph most of the time, but when the file size between 50MB and 80MB, the performance of Ceph and Hadoop is close. 33

44 Figure 4.4: Read small file In this case, we uploaded 16GB file in total, and split the file from 100MB to 1000MB; we wanted to test the effect of performance of data transport between larger file and small file. From the results, we can observe that the performance of Hadoop is better than the other two. Moreover, when the file size is smaller than 500MB, Ceph is better than GlusterFS, but if the file size is larger than 500MB, Ceph is either similar to GlusterFS or even worse. 34

45 Figure 4.5: Write small file In this case, we download 16GB file in total, and split the file from 100MB to 1000MB. From the result, we found that the performance of them is close, but Ceph rises suddenly when the file size is between 500MB and 600MB. And Hadoop is the best in them. Figure 4.6: Read small file 35

46 4.3 User Interface of ViPR ViPR can integrate some different original storage architecture through a data service, and provide some automatic mechanism for the storage systems to simplify the management work and decrease the work complexity. It can transform the physical storage from different company or system to be a single, scalable and open virtual storage platform. It can manage all storage resource by a single interface, as shown in Figure 4.7. Figure 4.7: Dashboard As shown in Figure 4.8, people can register their storage system to the physical assets part. And people can add some computers in the physical storage array to be a host, as shown in Figure

47 Figure 4.8: Add PhysicalAsset Figure 4.9: Add Host 37

48 ViPR transforms the physical storage to be a virtual storage plane, and it can create or manage the virtual storage pools automatically by the policy from the manager. When people create the policy, he can set how to create the virtual storage pool according to the storage performance which the service needs, as shown in Figure 4.10 and Figure Figure 4.10: Add VirtualAsset 38

49 Figure 4.11: VirtualPool Create There are some data service will run on the plane, and the virtual storage pool and virtual storage array can provide the application with the data service, as shown in Figure People can split the virtual storage pool to be some virtual storage arrays, and manage them by the policy. There are three types in the data service: Object: the file systems can store, take a file and modify the unstructured data by the data service, and the origin data application system can read or write the data easily. HDFS: collocation the service, people can transform the origin storage device to be a big data storage place. And they can use it to build a big data environment quickly without building an analysis cluster. Block: in the block data service, people can take a data across several storage array, and it can support many kinds of applications, file systems, database and virtual platforms. 39

50 Figure 4.12: DataService Setup 4.4 Discussion We used ViPR to provide the storage management to system; it can manage storage array simply through the data plane and control plane. The data plane is an advantage data service which runs on storage arrays, for example it can offer ability of object storage on file system storage platforms or offer ability of block storage on normal storage platforms. The data service which can virtualize the storage system combines different data types (file, object, and block), protocol (iscsi, NFS, and REST), security, usability. We used ViPR to offer a storage pool to our system, and we prove that the system will run normality by the experiments in this chapter. According to the experimental results, we can find that the performance of Hadoop HDFS is better than the others, and thus, Hadoop HDFS is more mature and has higher performance than the others. And ViPR can create virtual storage pool and assign the storage space to different kinds of storage. 40