Rackspace Private Cloud Reference Architecture

Transcription

1 RACKSPACE PRIVATE CLOUD REFERENCE ARCHITECTURE SOLIDFIRE Rackspace Private Cloud Reference Architecture

2 SolidFire Legal Notices The software described in this user guide is furnished under a license agreement and may be used only in accordance with the terms of the agreement. Copyright Notice Copyright 2013 SolidFire, Inc. All Rights Reserved. Any technical documentation that is made available by SolidFire is the copyrighted work of SolidFire, Inc. and is owned by SolidFire, Inc. NO WARRANTY: The technical documentation that is made available by SolidFire, Inc. is delivered AS-IS, and SolidFire, Inc. makes no warranty as to its accuracy or use. Trademarks Rackspace is a registered trademark in the US and other countries. Reg. No. 2,510,081 1 Fanatical Place City of Windcrest San Antonio, TX 78218

3 Table of Contents Executive Summary...4 Goal...4 Audience...4 The SolidFire Advantage...5 Quality of Service (QoS)...5 Operational Simplicity...5 High Availability...5 Data Efficiency...5 Solution Overview...6 Hardware Selections...7 OpenStack Controller...7 OpenStack Compute Nodes...7 OpenStack Block Storage...7 Networking Hardware...8 Test Environment Configuration...8 SolidFire Cluster Configuration...8 Configuring a SolidFire Node...9 Creating the SolidFire Cluster...10 Creating a SolidFire Cluster Admin...10 Rackspace Private Cloud OpenStack Configuration Installing and configuring RPC environment...12 RPC Environment File...12 Network Layout...13 Physical Layout...13 Test Validation...14 Test Objective...14 SolidFire Block Storage Integration Test Results...14 Provisioning Volumes with Guaranteed Performance...14 QoS Validation...15 Defining Volume Tiers...15 Creating the Volumes...15 Attaching the Volumes to the OpenStack Instances...16 Validating Performance...16 IO Scheduler Optimization...18 Conclusion...18

4 Executive Summary Traditional storage systems were not designed for the unique challenges presented by large-scale cloud infrastructure. In a cloud environment, the performance, scale and management requirements are different from traditional enterprise settings. A SolidFire storage system is architected specifically to address these issues. With deep integration into OpenStack s block storage services, users are able to take advantage of SolidFire s unique ability to deliver isolated and guaranteed storage performance to each virtual machine running on their private cloud infrastructure. This document will outline the architecture, and related configuration guidelines, necessary to run SolidFire as the back-end block storage system with the Rackspace Private Cloud Software v4.2. This document will also demonstrate how to provision precise levels of storage performance and capacity from the SolidFire system to any sized virtual machine. Goal This document has three goals; 1) Outline the configuration of the Rackspace Private Cloud software with SolidFire storage 2) Validate integration with the OpenStack Block Storage service, and 3) Demonstrate SolidFire s ability to deliver guaranteed storage performance to hosted virtual machines. Audience The target audience for this document is Solutions Engineers, Architects, Consultants, and IT administrators looking to understand how to configure and leverage SolidFire storage in a Rackspace Private Cloud. This document assumes that the reader is familiar with OpenStack specific concepts as well as the basics of SolidFire architecture. Additional documentation on OpenStack and SolidFire can be found at: OpenStack: install/apt/content/example-installation-architecture.html installing-openstack-with-rackspace-private-cloud-tools SolidFire: rackspace-private-cloud-installation-prerequisites-andconcepts#hardware-prereq Configuration.pdf 4

5 The SolidFire Advantage The SolidFire storage system was been built specifically to address the storage challenges encountered in a cloud infrastructure. The goal was to address the limitations of traditional storage that make it difficult to deliver consistent, predictable performance in a shared infrastructure. Key benefits of using SolidFire in a Rackspace Private Cloud include: Quality of Service (QoS) SolidFire has architected hard QoS controls into the system that are defined in terms that actually mean something to a customer, IOPS and GBs Each volume is configured with minimum, maximum, and burst IOPS. The minimum IOPS provides a guarantee for performance, independent of what other applications or tenants on the system are doing. The maximum and burst settings control the allocation of performance, ensuring consistent performance all tenants on the system. Operational Simplicity The SolidFire storage appliance is designed and built for orchestration platforms like OpenStack. Operationally, the SolidFire system is an easy to configure and fully automated solution that can scale horizontally as your Rackspace Private Cloud grows. With regards to SolidFire s OpenStack integration, minimal adjustments are required for the system to interoperate with OpenStack. The configuration on the Cinder node is simple and consists of just four entries in the cinder.conf file. There are no extra libraries or special packages to install on your cinder nodes, compute nodes or your SolidFire cluster. SolidFire offers a full and complete easy to use REST based API, making full automation of your storage a reality. All features and functions that are available from the system UI are available as API calls. Through the OpenStack driver an administrator has the ability to efficiently manage every aspect of the SolidFire Cluster. High Availability By leveraging a clustered approach, data stored on SolidFire is automatically distributed in the background across all nodes in the cluster. This evenly balanced data layout provides built in data protection. RAID-based systems are notorious for causing SSDs to wear out prematurely due to their high write amplification. SolidFire s Helix data protection incorporates many endurance management techniques automatically and simultaneously including: Multiple distributed copies of each block of data for built-in high availability Balanced wear distribution across all SSDs to increase drive life Automatic self healing processes to overcome hardware failures Data Efficiency SolidFire s approach to data reduction makes it an extremely efficient storage option for private cloud environments. All data is compressed and deduplicated in-line, without impacting performance, to minimize storage footprint. All remaining free space is thin provisioned and made available to OpenStack for provisioning. By thin provisioning the dataset, SolidFire effectively eliminates stranded storage. When you re ready to grow you re SolidFire cluster, simply add a node or nodes to your cluster. There is no need to modify your OpenStack Block Storage configuration or restart any services. 5

6 Solution Overview Figure 1 Figure 1 shows a topological view of the test environment. The Rackspace Private Cloud software consisted of the following core components networked to the SolidFire storage system. Controllers Compute One OpsCode Chef server Two OpenStack controller nodes (configured for HA) Keystone Horizon Glance Cinder Queues (with RabbitMQ) Databases (with MySQL) Chef-Server Controller 1 Controller 2 1GbE Mangement Network Compute 1 Compute 2 Compute 3 Three compute nodes (minimum config) 10GbE Storage Network SolidFire Cluster 6

7 Hardware Selections The following section outlines the hardware used in the test environment. OpenStack Controller The highly available OpenStack Controller was implemented on the following hardware: OpenStack Block Storage In this design, four (4) SolidFire SF3010s were used for OpenStack block storage. Below is a table outlining the hardware specs of the SF3010: Model Drives SolidFire SF3010 (x4) (10) 300 GB 2.5 SSD Servers Dell PowerEdge R710 (x2) System Memory / Read Cache 72 GB Shared RAM 48GB Write Cache 8GB Non-Volatile DRAM Processor Intel Xeon Processor (2) 2.5GHz 15m cache 6 core Local Disk 146GB (x2), RAID 1 Networking (2) 10GbE SFP+ iscsi (2) 1GbE RJ45 management Network Broadcom NetXtreme II (4-port) 1GbE, Intel 82599EB 10GbE (2-port) Power Supply (2) Hot-plug redundant highefficiency 750W OpenStack Compute Nodes The OpenStack compute nodes in this design were built on the following hardware: Average Watts Enclosure Rack support 150W to 450W, depending on IO load 1RU Height: 42.8mm (1.7 ) Width: 434mm (17.09 ) Depth: 731mm (28.8 ) 4 post rack, tool-less sliding rails Servers Dell PowerEdge R710 (x3) Weight 17.2 kg (38 lbs) RAM 48GB Processor Intel Xeon Local Disk 300GB (x2), RAID 1 Network Broadcom NetXtreme II 1GbE (4-port), Intel 82599EB 10GbE (2-port) 7

8 Networking Hardware In this design we used one (1) 10GbE Ethernet switch for the storage network and one (1) 1GbE switch for management network. Note that a highly available production deployment would require redundant switches for each network. Arista 7050S 10GbE switch Total 40GbE Ports: 4 (QSFP+) Test Environment Configuration For the configuration of the test environment we will specifically call out where the configuration differs from the recommendations provided by both the Rackspace and SolidFire documentation. There were four specific configuration differences to be noted including: Total 1/10GbE Ports: Latency: Typical Power Draw: 64(SFP+) ns (SFP+) ns (QSFP+) 125W SolidFire Cluster Configuration Rackspace Private Cloud Software Configuration Network Layout Physical Layout Cisco Catalyst 2950 series 1GbE switch Total # of 1GbE Ports: 24 All nodes on the 10GbE network were connected using Twinax cables to the 10GbE Arista switch. The nodes on 1GbE were connected using Ethernet cables to the 1GbE Cisco switch. SolidFire Cluster Configuration SolidFire s Element Operating System (Element OS) is designed to deliver all of the functionality required to manage and maintain application performance within a large multi-tenant cloud infrastructure. For the test and validation effort we tested on Element OS version 5. More on the SolidFire Element OS can be found here: solidfire-element-os/ An Intel 82599EB 10-Gigabit Network interface card and a NetXtreme II BCM5709 Gigabit Ethernet card were used on the Dell PowerEdge R710s. Intel 82599EB 10-Gigabit: Management of the storage system is done via the SolidFire web User Interface (UI) or API. The Terminal User Interface (TUI) is leveraged for initial network setup and Element OS upgrades. This section will outline the procedures necessary to configure the SolidFire nodes and then create the storage cluster. # of Ports: 2 Requirements for Cluster Configuration: Data Rate Per Port: 10Gbps An available 1GbE network for management traffic System Interface Type: PCIe v2.0 (5.0GT/s) NetXtreme II BCM5709 Gigabit Ethernet # of Ports: 4 Data Rate Per Port: 1Gbps An available 10GbE network for data traffic IPs allocated for all of your SolidFire nodes on the 1GbE and 10GbE networks, plus one additional Management virtual IP for each network. SolidFire nodes with Element 5 OS installed. If Element 5 OS is not on the nodes, contact SolidFire support. System Interface Type: PCIe v2.0 8

9 Configuring a SolidFire Node SolidFire nodes require initial network configuration before they can be accessed by the web UI or API. This configuration is done via the TUI. Once initial network configuration of a node is complete, it can be accessed via a single highly available Management IP address to configure a cluster. To configure the network on a SolidFire node, simply plug into the physical node itself to access the Terminal User Interface (TUI). Screenshot: SolidFire Web User Interface From the web UI click on the Cluster Settings tab to define the storage cluster. Screenshot: SolidFire Terminal User Interface The 1GbE address will be displayed in the Address field if a DHCP server is running on the network, otherwise a static IP address can be set at this time. In this design, a DCHP server was running on the network and the IP addresses were assigned automatically. This IP address is used to remotely access and configure the cluster in the Element Web UI. Once an IP address has been set, navigate to the Web UI by typing in the IP address of the node with port 443 into a web browser. Screenshot: SolidFire Cluster Settings The Hostname field is the node s hostname, and the Cluster field is the name of the cluster to which this node will belong, both of these names are arbitrary. Fill in these two fields, and then save. Repeat this process for all nodes in the cluster, making sure that the nodes in the cluster all have the same value for the Cluster field. Examples: 9

10 Creating the SolidFire Cluster Once the nodes have their network and cluster information saved, a cluster is ready to be created. A cluster can be configured from any of the SolidFire nodes. Simply navigate to a node by typing its IP address into the browser. Examples: A screen similar to the screen shot below should appear: Screenshot: Cluster Summary Creating a SolidFire Cluster Admin Before building the OpenStack cluster, a cluster admin account must be created for OpenStack in the SolidFire management interface. Navigate to the management interface and click on the Cluster Admin tab. The following is explanation of the fields for creating a new cluster: Management IP: Routable virtual IP on 1GbE or 10GbE network for admin tasks. Example: ISCSI VIP: Virtual IP on 10GbE network for iscsi discovery. Example: Data Protection: Two-way data protection: Always on Create Username: The user name for authenticated login to the cluster. Entered once and stored. Create Password: The password for authenticated login to the cluster Repeat Password: Standard password confirmation On the right, you will see all of the discovered nodes on the network. These are the nodes that are available to be added to the cluster. Fill in the fields, select all the nodes to be in the cluster, and press Create cluster. Note: A cluster can also be created using the API. See the SolidFire API guide at: Screenshot: SolidFire Cluster Settings Click on Create New Cluster Admin to create a new cluster admin with a username and password. Underneath the section Access Settings, check the boxes Reporting, Volumes, and Accounts. These credentials will be used when configuring the OpenStack Cinder services. All configurations to make OpenStack work with the SolidFire cluster are added into an environment file that RPC uses to set OpenStack s service configurations. This is covered in the, Rackspace Private Cloud OpenStack Configuration section. 10

11 Rackspace Private Cloud OpenStack Configuration Rackspace Private Cloud (RPC), requires at least one controller node, one compute node, and one node to run the chef-server. The prerequisites can be found in the Getting Started Guide : rackspace-private-cloud Dell PowerEdge R710s Chef-Server Role [single-controller] role [cinder-all] role [single-compute] In this design, a Dell PowerEdge R710 was used for the chef-server, the compute nodes, and the controller nodes. See Hardware layout for hardware specifications of the nodes. RPC v4.2 uses Opscode Chef to deploy OpenStack Havana. One node must be installed as a chef-server and every node in the OpenStack cluster will then be assigned a role using the chef-server. Each role assigned to the nodes is specific to that node s function in the OpenStack cluster. For example, the node that has been designated the controller node will have the role single-controller. The node that has been designated the compute node will have the role single-compute. The OpenStack cluster uses OpenStack s Cinder-API in combination with SolidFire s Cinder driver to build and manage volumes. This means that one node in the OpenStack cluster must be chosen to host the Cinder services and configured to use the SolidFire Cinder driver. This node can be one designated node used to run the only the Cinder services, or the Cinder services can be run on any node already running other OpenStack services, such as the controller or compute node. In this design, the Cinder services are run on the controller node. The following diagram is an abstract of the OpenStack environment with the SolidFire storage cluster attached as the back-end storage device. Note that this diagram does not show an HA environment. 1GbE Switch Internet 10GbE Switch Each SolidFire Node is connected SolidFire SF3010 SolidFire SF3010 SolidFire SF3010 SolidFire SF3010 SolidFire Cluster 11

12 Installing and configuring RPC environment A detailed guide on installing RPC can be found here: installing-openstack-with-rackspace-private-cloud-tools In summary, the following points can summarize the installation: Prepare the Nodes Install Chef Server Install RPC cookbooks Install chef-client on all the nodes in the OpenStack cluster. Create an Environment File Set node environment Run Chef-Client on the OpenStack nodes Follow the RPC guide to install OpenStack, but change your environment file to account for the SolidFire Cinder driver configurations. This needs to be done before running chef-client on the OpenStack nodes. RPC Environment File The RPC chef environment is where the SolidFire Cinder driver is configured. To configure Cinder to use SolidFire, you must set the following variables in the Chef environment: node[cinder][storage][provider] = solidfire By default, this variable is unset and Cinder uses LVM. Specifying solidfire ensures that Cinder-volume will use the SolidFire API instead. After you set this variable, you must specify the following additional variables with the values appropriate to the environment: node[ cinder ][ storage ][ solidfire ][ mvip ] = <servicevipforsolidfiredevice> node[ cinder ][ storage ][ solidfire ][ mvip ] = username node[ cinder ][ storage ][ solidfire ][ mvip ] = userpassword An example of the relevant section in the environment file is below. override_attributes : { cinder : { storage : { provider : solidfire, solidfire : { mvip : , username : <your_username>, password : <your password> } } } }, After making the change to the environment file, run chefclient on the OpenStack nodes. Make sure to run chef client on the controller node first. Test the configurations by creating a volume via the cinder API or the dashboard, then verifying if the newly created volume is displayed on the SolidFire management interface. 12

13 Network Layout SolidFire uses virtual IPs to represent a single logical group. These virtual IP address present a consistent interface regardless of the number of nodes in the SolidFire cluster. Two networks are needed; one 1GbE network and one 10GbE network. IPs are needed for the chef-server, each OpenStack node, and each SolidFire node. 2 additional IPs are needed for the Management Virtual IP(MVIP) and Storage Virtual IP (SVIP). MVIP is the 1GbE management virtual IP and SVIP is the 10GbE storage or iscsi virtual IP. Below is a diagram of the network layout. Chef-Server Controller Node 1GbE IP: GbE IP: Compute Node 1GbE IP: GbE IP: The chef-server, the OpenStack nodes, and the SolidFire nodes are on the same 1GbE network connected by switch. The OpenStack nodes and SolidFire nodes are on the same 10GbE network connected by a switch. Note that it s not necessary for the chef-server to be on the same 10GbE network. Physical Layout Below is a diagram of the rack used for this design. The rack consists of a 1GbE switch, a 10GbE switch, two Dell PowerEdge R710s, and the four (4) SF3010 SolidFire nodes. Each SolidFire node has one 1GbE and one 10GbE interface plugged into their respective switches. 1GbE Switch network 10GbE Switch network SolidFire SF3010 1GbE 1P: GbE 1P: SolidFire SF3010 1GbE 1P: GbE 1P: SolidFire SF3010 1GbE 1P: GbE 1P: MVIP: ISCSI VIP: SolidFire SF3010 1GbE 1P: GbE 1P:

14 Test Validation Test Objective The first objective of the test was to validate full SolidFire driver and system compatibility with the Cinder Block Storage service. Additionally, we wanted to demonstrate advance provisioning capabilities by defining and verifying SolidFire s guaranteed QoS of service feature. SolidFire Block Storage Integration Test Results Testing showed that all Cinder functionalities were available and functioned properly. A list of the Cinder functions and test results are presented in the table below. List volumes Provisioning Volumes with Guaranteed Performance The SolidFire volume driver for the OpenStack Cinder block storage service enables direct access to the underlying functionality in the SolidFire storage system. By configuring the SolidFire Quality of Service (QoS) features with Cinder, users of Rackspace Private Cloud can provision block storage volumes with a SolidFire guaranteed level of performance. Configuring volumes with QoS in Cinder is achieved by creating volume types with extra specs. A volume type in Cinder is similar in concept to compute flavors, where each volume type represents a set of volume properties. Once a volume type has been created it can be assigned vendor specific attributes known as extra specs. The extra specs associate storage system attributes to the volume type and determine which storage system (back-end) to use for provisioning. For SolidFire the extra specs are used to define the minimum, maximum and burst IOPS settings of a volume. Create volumes List volume detail List volume type List single volume detail Delete single volume List single volume type Create snapshot List snapshot entities List detailed snapshot When the request to provision storage occurs, the SolidFire driver in OpenStack will check the volume type and extra spec information by default. Define a Volume Type cinder --os-username <username> --os-password <password> type-create <type name> Define a Volume Type Extra Specs cinder -os-username <username> --os-password <password> type-key <UUID> set qos:miniops=<min IOPS Value> qos:maxiops=<max IOPS Value> qos:burstiops=<burst IOPS Value> View all information about single snapshot Delete single snapshot Create volume type Delete volume type 14

15 QoS Validation To validate SolidFire s ability to guarantee storage performance to virtual machines running in a Rackspace Private Cloud. We performed the following tasks: 1. Define three (3) disk volume tiers using volume types and extra specs 2. Create three (3) volumes backed by SolidFire s QoS 3. Attach the volumes to the OpenStack instances 4. Use VDBench to generate a workload to the volume and measured the IOPS to ensure each volume was running at their specified QoS level. Defining Volume Tiers Three different volume types were defined in order to test three different sets of QoS values. The volume types created were named Gold, Silver, and Bronze. The Gold volume type was given the highest QoS values, the Silver volume type was defined with Medium QoS values, and the Bronze volume type had the lowest QoS values. Assigned QoS Values: 1. Gold Min=1000; Max=2000; Burst= Silver Min=500; Max=1000; Burst= Bronze- Min=250; Max=500; Burst=1000 The volume types were created by running the following commands on the OpenStack controller node. Keep in mind that the authentication data is stored in environment variables at this point. $>cinder type-create gold; $>cinder type-create silver; $>cinder type-create bronze; After creating the volume types, QoS values need to be assigned using cinder extra-specs. $>cinder type-key <UUID_of_gold_type> set qos:miniops=1000 qos:maxiops=2000 qos:burstiops=4000 $>cinder type-key <UUID_of_silver_type> set qos:miniops=500 qos:maxiops=1000 qos:burstiops=2000 $>cinder type-key <UUID_of_bronze_type> set qos:miniops=250 qos:maxiops=500 qos:burstiops=1000 After running these commands the three volume types with QoS values have been created. Creating the Volumes The following commands were used to create the bootable volumes. >$ cinder create --display-name goldvol -volume-type gold 100 >$ cinder create --display-name silvervol -volume-type silver 100 >$ cinder create --display-name bronzevol -volume-type bronze

16 Attaching the Volumes to the OpenStack Instances After creating the volumes, the volumes were attached to their respective instances with the device name /dev/vdb. The following command was used to attach the volumes. GoldVM results As expected, the attached gold volume showed the highest IOPS. After running the vdbench workload mentioned above, the volume maxed out it s specified effective max bandwidth, and performed at a consistent rate of 2000 IOPS. >$ nova volume-attach <instance_id> <volume_id> /dev/vdb Validating Performance In this section, vdbench was used to run workloads on the three different types of volumes that were created above. The follow parameters were used in vdbench: data_errors= hd=default,user=user,shell=vdbench hd=hd1,system=localhost sd=default,openflags=o_direct,range=(1,100) sd=sd1_1,host=hd1,lun=/dev/vdb,openflags=o_direct wd=default wd=wd0,readpct=80,seekpct=100,xfersize=(4k,100),sd=sd1_1 rd=default,iorate=max,elapsed=600,interval=1,forthreads=16 rd=rd1,wd=(wd*) 16

17 SilverVM results The silver volume also maxed out its effective max bandwidth, performing at an average of 1000 IOPS. BronzeVM results The bronze volume had the lowest average IOPS of 500. This is expected since the bronze volume had the lowest set of QoS values. Nonetheless, it still maxed out its effective max bandwidth. 17

18 IO Scheduler Optimization While running the tests, it has been observed that maximum performance is gained when using the NOOP IO scheduler. The NOOP scheduler implements a simple queue for all incoming I/O requests, without re-ordering and grouping the ones that are physically closer on the disk. This scheduler is ideal for SDDs, with no mechanical parts there are no physical movements taking place within the disk. To change the default IO scheduler to be persistent across reboots, make the following change to the /etc/default/grub file: GRUB_CMDLINE_LINUX_DEFAULT= splash quiet elevator=noop To change the IO scheduler of a specific device on a live environment run the following command: Conclusion By combining SolidFire s deep OpenStack integration and QoS feature set with the simplicity of a Rackspace Private Cloud, users have the ability to build reliable, scalable private clouds designed to deliver consistent and predictable performance to 10s, 100s or 1000s of applications in parallel. With this solution backed by Rackspace s world-class support organization, users can feel confident with the performance and reliability of their private cloud infrastructure. The reference architecture in this document simplifies the deployment of these products by providing infrastructure guidelines and best practice configuration. If you have any questions regarding the steps and/recommendations outlined above, or would like to see a live demo, please contact us at sales@solidfire.com. $> echo noop > /sys/block/<device>/queue/scheduler 18

19 SOLIDFIRE 1620 Pearl Street, Suite 200 Boulder, Colorado Phone: About SolidFire SolidFire is the market leader in high-performance data storage systems designed for large-scale public and private cloud infrastructure. Leveraging an all-flash scale-out storage architecture with patented volume-level Quality of Service (QoS) controls, providers can now guarantee storage performance to thousands of applications within shared infrastructures. By using real-time data reduction techniques and system-wide automation, SolidFire is fueling new and profitable block-storage services that are advancing the way the world uses the cloud.