Private Distributed Cloud Deployment in a Limited Networking Environment

Private Distributed Cloud Deployment in a Limited Networking Environment Jeffrey Galloway, Susan Vrbsky, and Karl Smith The University of Alabama jmgalloway@crimson.ua.edu, vrbsky@cs.ua.edu, smith102@crimson.ua.edu Abstract The focus of our research began with the deployment of a cloud computing system to offer resources similar to commercial vendors. Realizing the implications of deploying such a system in an environment with limited networking resources (IP addresses), we decided to find a solution that would work giving our cloud only one public IP address. Users will be able to access cloud resources through a simple web interface and maintenance of the cloud will be contained with private networking resources. Users needing resources in the local cloud will rely on a NAT router, with other functionalities, to gain access these resources. We will also demonstrate, if the need arises to have multiple geographically distributed clusters in the local cloud, how to scale in this sense with only one IP address per cluster. Index Terms Cloud Computing, IP Networking, Resource Distribution. T I. INTRODUCTION HE cloud computing architecture is becoming a dominant contender in the distributed systems paradigm. Its differences from the client/server architecture are based in its heavy use of resource elasticity. The cloud architecture uses virtualization technology in distributed data centers to allocate resources to users as they need them. Cloud computing has emerged from the previous industry standards, such as grid and cluster computing. Although cloud architectures are similar to these distributed systems, the resources are usually maintained by a single entity and might not be used by customers to complete similar goals. Depending on the level of control the user has, Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS), or Software-as-a-Service (SaaS), jobs can differ greatly from customer to customer [7]. There are many implementations of the cloud architecture by various commercial vendors and open source communities. The Amazon Cloud, or collectively known as Amazon Web Services (AWS) is arguably the leader when it comes to offering customers access to hardware in the cloud (IaaS). To demonstrate this commanding lead in the field, the vendor, Netflix, Inc. (the largest on-demand movie and television streaming company) uses AWS to run mission critical customer-facing and backend applications [2]. Microsoft has entered the cloud computing paradigm with its Azure platform (PaaS) [3]. This platform allows customers to create and run Contact Author: Jeffrey Michael Galloway Conference: ICOMP applications hosted by Microsoft s datacenters. Customers using Azure have access to Microsoft s cloud operating system and all of the services needed to develop applications remotely. Unlike AWS, Azure does not allow the user lower level access to hardware. Google s Docs application [4] allows users to store and edit office document related data. Users of this application have no permissions to access underlying hardware, or change the software applications presented to them (SaaS). The user consumes this resource with little or no care of how it s implemented. In this paper, the cloud architecture will refer specifically to Infrastructure as a Service (IaaS), providing users the ability to create virtual machines. Following in the footsteps of cloud leaders such as Amazon and Google, open source communities have provided software packages that allow individuals to deploy their own local cloud. As customers become more and more dependent on retrieving data whenever they want it, they become heavily dependent on the cloud vendor s reliability for data access. There could be many reasons for a user or organization to deploy a local cloud. One example would be the user wanting a higher utilization of their physical resources. Another example could be that some data created by local organizations cannot be stored by a public cloud vendor. Whatever the reason for a local cloud, users still need access to the data stored, which requires expensive networking resources. When a user needs to connect to a local cloud resource, each instance of that resource has to have a unique private IP address. In the case of our experiments, the resources are stateless virtual machines. The user is given a list of operating systems to choose from, and if persistent storage is needed, they have the ability to mount space located on the storage controller. The open source community is led by three prominent distributions: Eucalyptus, OpenNebula, and Nimbus [1]. Our local cloud implementation uses Eucalyptus. An overview of the Eucalyptus architecture will be given in section three. The topic of this paper is concerned with deployment of privately maintained clouds using limited networking resources. The rest of the paper is organized as follows. Section two will present the problem statement of deploying a private cloud into an environment with limited networking capabilities. Section three explains how to build the cloud using Eucalyptus. Section four gives details on deploying cloud resources with minimal interactions from users. Section

five shows how to deploy a distributed multi-cluster setup using VPN. Section six presents our conclusions. II. PROBLEM STATEMENT Customers may be reluctant to move their data to off-site cloud vendors due to access regulations, cost, and trust issues [9]. Instead, new software has become available for them to build their own personal cloud computing system. This gives the customer the ability to understand the issues and benefits of using cloud technology for distribution of resources before making the move to an enterprise vendor. There is a major problem with deploying a cloud locally; network resources in terms of IP addresses are usually expensive, and therefore, limited for smaller organizations. We are proposing a solution that needs only one public IP address for resource distribution to users of the local cloud. This approach will provide benefits of more than just efficient use of network resources. By having only one public address associated with the cloud system, security vulnerabilities are decreased. This is due to the fact that all incoming and outgoing traffic to the cloud will be associated with one IP address. This address can be monitored more efficiently than if the entire cloud system were implemented on public IP addresses. In the case of our cloud construct proposition, no new network infrastructure will be needed in the organization. In some cases, organizations may have resources geographically distributed. The idea of implementing a cloud architecture across these resources may be a possibility if a minimum of networking resources are used to connect them. It is possible to scale an organization s local cloud to other sites using VPN and a single IP address per distributed cluster. We understand that as demands rise to access resources not located in the local cloud, a bandwidth bottleneck may arise. Implementation of this architecture should give careful attention to the bandwidth requirements of the local cloud WAN IP address, and the bandwidth needed to connect each distributed resource to the local cloud. III. LOCAL EUCALYPTUS CLOUD Currently, our local cloud, fluffy, consists of seven Dell OptiPlex desktop computers, one network attached storage device, one 1Gbps Ethernet switch, and one Linksys WRT54GL NAT router. The underlying operating system of our cloud is Ubuntu Server 10.10 64-bit, which includes the Eucalyptus 2.0 open-source cloud architecture software [5]. Following the current specifications of deploying Ubuntu Enterprise Cloud (UEC) [8], we have configured the following machines to make our local cloud environment. Our goal is to deploy this cloud environment into a working environment with a single IP address. The following sections outline the main components of our local cloud. We make note here to show that our cloud consists of nonhomogeneous commodity hardware that could easily be found in office environments. The only equipment that requires specific consideration are the servers used to run node controller software. These servers should have CPU s that are capable of hosting virtual machines. A. Cloud Controller This machine is the front end user interface to our cloud setup. It is hosted on a Dell OptiPlex 745 desktop (Core 2 Duo 2.8 GHz, 2GB RAM, 80GB hard drive). Its purpose is to provide the web interface to users while interacting with the rest of the components in the cloud architecture. The cloud controller monitors the availability of resources of various components in the local cloud and monitors the running instances currently deployed on the node controllers. This device is fully customizable in the fact that we can install additional software packages to meet our needs. For example, we have apache, MySQL, Java, and PHP-CLI installed to handle various processes of deploying our cloud resources. B. Cluster Controller/Storage Controller The current deployment of UEC requires the cluster controller and storage controller to be on the same physical machine. These are hosted on a Dell OptiPlex 755 (Core 2 Duo 2.8 GHz, 2GB RAM, 160GB hard drive). The cluster controller determines on which node controller a deployed virtual machine will run. It also allows network access to the virtual machines running on the node controllers. DHCP and DNS processes for the virtual machines are maintained on this controller. The cluster controller is responsible for load balancing virtual machines across all node controllers. In the case that there are multiple node controller clusters, each cluster will be led by an individual local cluster controller. All of the cluster controllers report back to the cloud controller on the status of the node controllers that it controls. The storage controller gives persistent storage access to instance users. This service is similar to the elastic block storage service from Amazon Web Services. Storage blocks can be mounted to running instances since the virtual machine concept in Eucalyptus is stateless. Our current setup has only one cluster, therefore we have only one cluster controller. Alternative cloud configurations will show later that it is possible to connect many resources at different locations back to the cloud controller using only one IP address per cluster. C. Walrus Storage Controller Our walrus controller is hosted on a Dell OptiPlex 620 (Pentium D 2.93 GHz, 2GB RAM, 120GB hard drive). The walrus controller stores machine images that can be launched as virtual machines in the local cloud. We are currently hosting two versions of Ubuntu, 9.10 and 10.04 for users to access. The walrus controller will hold any machine image that we wish to make available to users. When the user decides to launch a virtual machine, the machine image has to be transferred from the walrus controller to the node controller (Section III.D) before being booted. If a specific image from the walrus is being used relatively often, that image can be cached on the node controller to decrease the amount of traffic in the cloud s private network.

D. Node Controller The node controller is a machine with VT (Intel) extensions on the CPU(s) used to host the running virtual machines in the local cloud. Our current setup has three node controllers. The first node controller is hosted on a Dell OptiPlex 755 (Core 2 Duo 2.8GHz, 2GB RAM, 160GB hard drive). Our second node controller is hosted on a Dell OptiPlex 960 (Core 2 Duo 2.93GHz, 2GB RAM, 160GB hard drive). The third node controller is a custom build (Core 2 Quad 2.5GHz, 4GB RAM, 320GB hard drive). The default hypervisor used by Eucalyptus is KVM. We are satisfied using this hypervisor because it is a full virtualization solution for Linux operating systems on x86 hardware containing virtualization extensions (Intel VT or AMD-V). The default configuration settings located in: /etc/eucalyptus/eucalyptus.conf defines a variable named MAX_CORES. This variable is initially set to the number of physical cores located on the specific node controller. Administrators can change this initial value to allow for a higher number of virtual machines per node controller. The equation below gives the maximum number of virtual machines KVM will allow on any single physical node controller: F. Client Device This device is a Dell OptiPlex 280 (Pentium 4 2.8GHz, 2GB RAM, 120GB hard drive). This machine is used for testing the cloud. Instances can be launched, and cloud components can be accessed by SSH. The router used in our setup is the Linksys WRT54GL. This device has been flashed with version 24 of DD-WRT [6]. This gives us more control over our network through the manipulation of ports through SSH connections, VPN access, and DHCP services. The open nature of the router will become important in the deployment of our local cloud. # of VM s = 8*(actual number of physical CPU cores) (1) The total number of virtual machines in our setup allows: 8*(actual number of physical CPU cores/2) (2) Equation (2) ensures that no single CPU will be overloaded (at most 4 virtual machines per physical core) and virtual machines will execute with sufficient processing resources. Other resources also play a role into the number of virtual machines available for launch. These include the total amount of RAM and available hard disk space. We are able to handle this amount of virtual machines per core because the specific Linux operating systems we offer are without graphical user interfaces. Figure 1 shows the information from the cloud controller as it keeps track of resources while virtual machines are instantiated and terminated. AVAILABILITYZONE fluffy 192.168.1.100 AVAILABILITYZONE - vm types free / max cpu ram disk AVAILABILITYZONE - m1.small 0024 / 0032 1 192 5 AVAILABILITYZONE - c1.medium 0022 / 0028 1 256 5 AVAILABILITYZONE - m1.large 0011 / 0014 2 512 10 AVAILABILITYZONE - m1.xlarge 0006 / 0007 2 1024 20 AVAILABILITYZONE - c1.xlarge 0002 / 0002 4 2048 20 Figure 1: Available resources on the local cloud. E. Backup Device The backup solution for our local cloud is a Dell OptiPlex 270 (Pentium 4 3.2GHz, 2GB RAM, 120GB hard drive). We use this device to run a scheduled backup on all other devices in our setup. The device mounts folders to our NAS (Netgear Stora 1TB), which stores our nightly backups. Figure 2: Local Cloud Architecture. As of now, users with private addresses have the ability to launch virtual machines in our cloud setup. This is an inconvenience since either the user has to be physically connected to our private network, or they need VPN accounts to access our cloud from a public address. A solution for making resources available to users outside of the private cloud network is given in the next section. IV. CLOUD INTERFACE WITH NETWORK LIMITATIONS In this section, we are proposing our cloud deployment scheme which allows an organization to relatively simply introduce a cloud infrastructure into their environment using a limited amount of networking resources. This deployment technique will take some initiative on the IT staff members depending on the amount of availability and scalability needed. Some things to take into consideration before deploying this type of cloud architecture (Figure 2) are WAN port bandwidth for single cluster setups and bandwidth between NAT routers while running the VPN protocol when deploying a distributed cluster setup. As previously stated, the resources given to users in our setup are virtual machines hosted on the node controllers. Users do not have physical access to the cloud s hardware resources, but can spawn virtual machines of different configurations to meet their demands. Users also have full root access to their instantiated virtual machines, and are responsible for any security precautions presented by having

resources available from potentially unsecured networks (WAN). The cloud architecture will be hosted completely within its own private network. A NAT router using DD-WRT firmware will be used to manage ports opened to the virtual machines created by users outside of the cloud s private network. The physical resources for the cloud would only demand the multiple servers needed, switches, and a single router. Users will access the cloud resources using commands to launch the virtual machines, and opened forwarded ports handled by the router. A. Router Setup and Log Information Our NAT router was loaded with DD-WRT firmware,giving us a great deal of control over its packet forwarding features. When connecting to these cloud resources, users shouldn t worry about the networking responsibilities of providing these resources. We have the ability to dynamically open and close ports and manipulate the iptables of our private network. This depends on the current number of virtual machines in execution. Port forwarding is necessary simply because we are depending on one WAN address to connect many users to many virtual machines that could be running similar protocols. The cloud controller keeps a list of the currently running virtual machines, as shown in Figure 3. This information will be used to maintain a log.txt file also located on the cloud controller. Each row in the log file is associated with an individual virtual machine. The only information required is the private IP address assigned to the virtual machine by the cluster controller, and the port forwarding information decided by the specific protocols the user chooses during launch. 192.168.1.151; protocol; port #; protocol; port #; protocol; port #, 192.168.1.152; protocol; port #; protocol; port #; protocol; port #, 192.168.1.200; protocol; port #; protocol; port #; protocol; port #; Figure 3: Log.txt file located on Cloud Controller. When a virtual machine is launched, the cluster controller informs the cloud controller of the pending network details. During the instantiation of the virtual machine(s) the graphical interface allows the user to select their preferred protocol. The default protocol used is SSH, which will initially only open port 22 on the virtual machine. All other ports will be closed, and therefore unreachable. Once the command has been sent to the cloud controller to boot a virtual machine, a process writes the private IP address of the new virtual machine to the log file. It then reads the log for protocols that are already used on other virtual machines, updates the log with the forwarded port for the user to utilize for each protocol, and notifies the user of these forwarded ports. Figure 4 shows the current status of the log file after a user instantiates a new virtual machine (151) with other virtual machines already running in the cloud. Once the log file has been updated for a specific virtual machine, the process communicates with the NAT router via SSH and updates the iptables. The following code shows how the cloud controller can update the iptables: $ iptables t nat l PREROUTING P tcp d $(nvram get wan_ipaddr) dport 1200 j DNAT to 192.168.1.151:22 $ iptables l FORWARD p tcp d 192.168.1.151 dport 22 j ACCEPT After the execution of these two commands, the router will forward incoming WAN packets from port 1200 to port 22 of the virtual machine with the IP address of 192.168.1.151. All protocols will be forwarded in this manner by the NAT router. When the user decides to terminate their virtual machine, a process on the cloud controller updates the log file by removing the record with the terminating virtual machine s IP address. Once this record is removed from the log, the cloud controller updates the iptables of the router by executing the following commands. The variable 1 is the record in the iptable specific to the terminating virtual machine s IP address: $ iptables D FORWARD 1 $ iptables D PREROUTING 1 t nat B. User Interface The user interface for connection to our cloud resources is relatively simple. The user should first create an account to use on the cloud. This is done by using the default Eucalyptus web interface. Once the user has an account, they must login and download the Credentials.zip file to use when connecting to virtual machines they instantiate. This file contains an RSA key, used to authenticate the user whenever they attempt to use resources on the cloud. As shown in Figure 5, the user navigates to the custom web interface that gives them the choice of operating systems, size of virtual machine (CPU, RAM, disk space), and protocols needed. Once their choices are made, they launch the virtual machine. The interface returns the IP address of that virtual machine and port forwarding values for the protocols they chose. The user then has the ability to connect to the virtual machine (typically by SSH) using their RSA key. From this point, the user has root access to perform any tasks they desire. 192.168.1.151; SSH; 1200; HTTP; 1500; 192.168.1.152; SSH; 1201; HTTP; 1501; FTP1; 1800; FTP2; 2000 192.168.1.153; SSH; 1202; FTP1; 1801; FTP2; 2001; SMTP; 2400 Figure 4: Example Log.txt file with instantiated vm s. Figure 5: User interface creation/deletion of vm s.

C. Network Considerations Deploying a scalable private cloud in the manner described in this paper requires careful planning to ensure availability to the cloud resources. The WAN connection will be the bottleneck when deploying this architecture. IT administrators should make sure to allocate as much bandwidth to the WAN port as possible. Generally, users will at least have SSH sessions open on each virtual machine. SSH, by protocol design, delivers constant data transmissions between the client and server processes to minimize attacks on traffic patterns. In our design, we used a 1Gbps copper Ethernet link to supply traffic to the WAN port. This provided sufficient bandwidth to host all of our available resources (32 virtual machines) to users. Another consideration that should be made is the selection of the NAT router. Our choice of the WRT54GL was one of low cost and ease of manipulating the firmware to allow root access. Even though we had a 1Gbps WAN link, our maximum sustainable throughput from WAN to LAN address was around 50Mbps. These speeds are typical of the 10/100Mbps Ethernet switch used by this router. This is enough to satisfy the current scale of our cloud, but would become a factor if a major increase in cloud resources were needed. Gigabit NAT routers such as the Linksys E4200 or Cisco RV220W provide excellent sustainable WAN to LAN throughput of 686Mbps and 720Mbps respectively. The implementation of the distributed multi-cluster setup described in the next section brings even more demanding requirements from the organizations available networking resources. network. The details of setting up VPN gateway tunnels to each router will not be covered as it is out of scope of this paper. Each cluster will have its own IP subnet, relaying this information back to the cloud controller. For example, the main cluster hosting the cloud controller would use 192.168.0.0/16 giving ~2 16 potential virtual machine IP addresses. Cluster 1 would use 192.169.0.0/16 giving the same amount of IP addresses for use as the main cluster. Each cluster would use this concept to provide a unique IP address across the entire geographically distributed cloud. Load balancing in a single cluster topology is executed by the cluster controller, but in this distributed setup, an additional layer of load balancing will be performed by the cloud controller. As the cloud controller is the only element that knows the state of the entire cloud, it will determine which cluster controller to contract when a user decides to launch a new virtual machine. V. DISTRIBUTED MULTI-CLUSTER SETUP Depending on an organization s geographical distribution, a multiple cluster setup may be desired. Each cluster in the Eucalyptus architecture contains the cluster and storage controller, and multiple node controllers. The DHCP service hosted on the cluster controller can be configured to accommodate unique IP subnets for each cluster. Using Eucalyptus default of IPv4, the private distributed cloud could potentially have on the order of ~2 32 virtual machines. IP addressing is clearly not a scalability factor when deploying this type of cloud. A. Router and VPN Setup The VPN protocol [10] will be used to deploy this cloud architecture. Each cluster is assumed to be geographically separate from all other clusters and from the main cloud controller cluster, as shown in Figure 6. In this architecture, we assume a main cloud cluster which contains the cloud controller, and auxiliary clusters used for hosting additional virtual machines. Each auxiliary cluster will require only one WAN IP address to connect back to the main cloud cluster. Users will still contact the single cloud controller to gain accounts and launch virtual machines. A VPN is an authenticated, encrypted connection between two networked devices. The WRT54GL NAT routers (and others) have the ability to form VPN gateway to gateway tunnels. This connection between NAT routers allows data to be transmitted over the internet as if it were in a single private Figure 6: VPN tunneling to connect distributed clusters. B. Scalability Considerations with a Distributed Architecture The potential for scaling using a distributed architecture ideally should be unlimited. Careful consideration should again be given to the available bandwidth to the WAN ports of each NAT router. The traffic generated in this setup is substantially greater than in the single cluster architecture. Organizations may consider using protocols such as ATM or others that guarantee a minimum bandwidth between NAT routers. The NAT router selected for this architecture is also more important than the one used in the single cluster architecture due to the increased demands on networking resources. Users will still use the single point of interface with the main cloud controlled cluster. There is no cloud controller in the auxiliary clusters, and therefore, no interface to use those resources directly. A future work direction to reduce the traffic and processing requirements of the single WAN port on the main cluster would be to propagate iptable updates to the other NAT routers of the auxiliary clusters. This way, users could connect directly to the resources using the single IP address of those auxiliary clusters.

VI. CONCLUSION This paper presents the idea of deploying a private cloud using a limited amount of networking resources. Organizations have the ability to create a private cloud and introduce it into their network without greatly affecting their current infrastructure. With the introduction of the single cluster cloud, users are able to request resources through a simple interface. Once the user has access to these resources, they can use it to perform any task they desire. When the user no longer needs the resources, they are released and can be used by others. We also introduced the concept of a distributed cluster architecture. In this architecture, VPN is used to connect the auxiliary cloud clusters back to the main cloud cluster. Each cluster is hosted on its own subnet of IP address, giving each virtual machine the ability to have a unique address. Scalability needs careful consideration while implementing the single or distributed cloud architecture. Administrators should be aware that the single WAN ports could be heavily utilized and should be allocated as much bandwidth as possible. This is especially true in the case of the distributed cloud topology. Administrators should also carefully consider the NAT router used when deploying this cloud. A few NAT router suggestions were given, and if more performance is needed, a dedicated NAT routing server could be used. REFERENCES [1] P. Sempolinski, D. Thain, A Comparison and Critique of Eucalyptus, OpenNebula, and Nimbus, 2 nd IEEE International Conference on Cloud Computing Technology and Science. 2010. [2] Amazon.com (2010, May 7). Netflix Selects Amazon Web Services to Power Mission-Critical Technology Infrastructure. Retrieved February 14, 2011 from http://phx.corporateir.net/phoenix.zhtml?c=176060&p=irolnewsarticle&id=1423977&highlight=netflix [3] Microsoft.com (2011) Windows Azure Platform. Retrieved February 14, 2011 from http://www.microsoft.com/enus/cloud/developer/default.aspx [4] Google.com (2011) Google Docs. Retrieved February 14, 2011 from https://docs.google.com/ [5] Eucalyptus Systems, Inc. Eucalyptus Open-Source Cloud Computing Infrastructure An Overview. Eucalyptus Systems, Inc. Web. February 14, 2011. Technical Document. http://open.eucalyptus.com/ [6] DD-WRT.com (2011) DD-WRT Open Source Firmware for NAT Routers. Retrieved February 14, 2011 from http://www.ddwrt.com/site/index [7] D. Chappell, A Short Introduction to Cloud Platforms an Enterprise- Oriented View. David Chappell & Associates. Web. August 2008. Sponsored by Microsoft Corporation. [8] Canonical Group Limited. An Introduction to Cloud Computing. Canonical Group Limited. Web. 2010. Technical Document. [9] W. Li, L. Ping, Trust Model to Enhance Security and Interoperability of Cloud Environment. CloudCom 2009. [10] Ravindran, R.S.; Changcheng Huang; Thulasiraman, K.;, "Managed Dynamic VPN Service: Core Capacity Sharing Schemes For Improved VPN Performance," Communications, 2007. ICC '07. IEEE International Conference on, vol., no., pp.211-216, 24-28 June 2007.