Securing Platform as a Service: A Technical Whitepaper on Security Practices at CloudBees As a consumer of cloud services, you are relying on your cloud service provider in ways that were previously limited to your own employees. The real or perceived control you have over employees accessing your sensitive information now includes your cloud service provider s employees in some manner. Second, your cloud service provider is offering its service to other users. Thus, your service provider s practices could potentially expose your sensitive information to other users. In this white paper, we provide details regarding security practices used by CloudBees and within our Platform as a Service (PaaS) to protect your sensitive data and guard against unauthorized access. The Java TM Paas Company
Securing Platform as a Service: A Technical Considerations Whitepaper for Continuous Security Integration Practices at in CloudBees the Table of Contents Executive Summary... 3 Background on the CloudBees PaaS... 3 Managing Credentials... 4 AWS Credential Management... 4 Credential Roll-Out... 5 Managing Security Around Remote Login and Development... 6 Remote Server Login... 6 Locked Down Access... 6 Backdoors... 7 Handling Problems... 7 Credentials and Password Policies... 8 Centralized Password Management... 8 OneLogin... 8 Password Resetting... 8 Keeping Credentials Private... 9 Email... 9 Conclusions... 9
Executive Summary Security should be a high priority for every company. The cloud changes existing security practices in two major ways. First, as a consumer of cloud services, you are relying on your cloud service provider in ways that were previously limited to your own employees. The real or perceived control you have over employees accessing your sensitive information now includes your cloud service provider s employees in some manner. Second, your cloud service provider is offering its service to other users. Thus, your service provider s practices could potentially expose your sensitive information to other users. In this white paper, we provide details regarding security practices used within CloudBees and in the CloudBees Platform as a Service (PaaS) to protect your sensitive data and guard against unauthorized access. When you place your trust in a cloud provider, you deserve direct answers to every question you have about security. If you are attempting to implement a private PaaS yourself, you will need to consider many of the solutions used by CloudBees already in our full service PaaS offering. In addition to this white paper, CloudBees offers even more detailed documentation of security practices under non-disclosure. Background on the CloudBees PaaS We have a detailed white paper (http://pages.cloudbees.com/cloudbees-platform-java-paas-wp.html) that documents the architecture of the CloudBees PaaS and some of the technical drivers behind its design. In that white paper, we also discuss security at a high level. The intent of this document is to provide details behind the security processes. Having a high-level understanding of the CloudBees PaaS, its functionality and its architecture is helpful to understand how we treat security. Figure 1 The CloudBees PaaS Architecture 3
As an end-user, when you interact with CloudBees, you do so through the GrandCentral console, the CloudBees API or the CloudBees SDK. Through these channels, you gain access to our hosted services for development and runtime. Behind the scenes, CloudBees also hosts our services platform, a set of shared services that interact with the hosted services you are using. These services, in turn, interact via a message bus with agents running in your targeted execution environment. Security concerns cross the entire platform. For example, we must limit access only to authenticated users within an account and allow them access only to the resources they are authorized to use. As the PaaS administrator, CloudBees itself requires access to resources you are prevented from accessing directly. Both services and agents running within our environment must be properly secured, as we manage resources on your behalf and on behalf of other users at the same time. These factors require a well thought-out security architecture as well as auditable processes. Managing Credentials In the cloud world, companies must be vigilant to the types of risk that exist when placing their code and infrastructure into the hands of others. Anytime you place your business data in the hands of a third party, there is risk. While providers like Amazon, Rackspace or VMware have built security credibility behind their names, inherently there is still some sort of risk involved. This is especially true with cloud providers, where the implementation and security behind the scenes is usually not visible to the end customer. You put your trust in the cloud provider, and the cloud provider owes you clear explanations and an ability to verify security practice. At CloudBees, we have a number of security measures in place to help safeguard your applications and code against external threats. We have honed some of these practices over the past few years. Others are best practices that everyone should be doing. Through our AnyCloud offering, the CloudBees PaaS is executing workloads on multiple infrastructure cloud providers. Today our public cloud offerings are primarily hosted on Amazon Web Services (AWS). In this white paper, we will use AWS as the focus of discussions, as it is likely to be more familiar to readers. However, the practices employed on AWS have mirrors in OpenStack or vsphere-based infrastructure cloud environments. AWS Credential Management CloudBees service offerings have been developed over the past few years and, like many others who have spent a few years in Amazon's cloud, we have evolved to take advantage of Amazon s improvements in credential management. Originally, Amazon offered one set of credentials that were universal across an AWS account. To tackle security, a lot of companies, including CloudBees, had multiple Amazon accounts for a layer of separation between services and access needs. In 2010, Amazon released a more fine-grained credential and access management system called AWS Identity and Access Management (IAM). Under the original AWS system, there was one centralized set of credentials for a specific AWS account. This meant that every developer who needed access had to be given these credentials. In addition, all of CloudBees core services that utilized the EC2 API also had to have these credentials distributed to the instances they ran on. Having a single set of credentials effectively a single key to the kingdom has some serious disadvantages. 4
If a developer left the company, this would necessitate a forced change of all of the credentials. Every developer would need to be given a new set of credentials, and every application would need to be updated to have the new set of credentials in place. No amount of automation makes this problem tractable at scale. When the number of CloudBees developers was small, and everyone did everything, everyone needed access. However, as more people joined the team and had different access needs, it became not only a security threat, but also just a development threat for everyone to have full access to all systems. Today the CloudBees access management system takes advantage of the Amazon IAM system. Even in an AnyCloud deployment, where CloudBees is managing workloads on a tethered cloud, access management is handled centrally by CloudBees. CloudBees uses both developer and service-specific credentials throughout our system. Not only can we have specific credentials now for each developer and each service, but we can also lock those credentials down to minimize security risks. One such example is our DNA service. This is an internal facing service that we use to monitor and manage instance and service health. The DNA application needs the ability to access instance lists, start and stop instances, and update IP address information, amongst other things. Not only does DNA now have its own credentials that are specific to its service, but those credentials are locked to a single fixed IP address. AWS will not accept commands using those credentials unless they originate from that single IP address. This approach helps minimize and contain any threat of DNA credentials being used maliciously. With developer-specific credentials comes the ability to much more easily rotate and disable access as needed. We can easily/quickly remove access for a specific set of developer credentials, without impacting other services or developers in the process. We can also limit developer access to the pieces of infrastructure they need in order to do their work. The concern here is not a rogue developer causing issues, but cases like a stolen laptop or even someone at a coffee shop seeing login credentials on the screen. Restricted access also limits what a developer can do accidentally, if they target the wrong thing or try something when they don't completely understand the potential outcome. Credential Roll-Out Rolling out new credentials to all developers and services is not easy. At CloudBees, this process has required considerable planning and execution, as any large enterprise already knows. For one, when people have restricted access to the system, they now are not able to react to major system issues that may creep up. When service access becomes limited, it can cause future potential issues. For example, if new features are added that make use of restricted API calls - nobody may remember they are restricted and a significant amount of debugging time may be spent trying to figure out why things don't work. You must also think through scenarios when developers still have access to change other developer or services permissions. For example, locking down our DNA service to a specific IP address increases security, but if any developer can go in later and change that lockdown, it may not be obvious that change ever happened. During some of our initial audits, we discovered security-related changes that were done ad hoc to quickly get something that had been broken working again, but then the security change was never later reversed. 5
As a result, part of our policy now is to disallow developers or services from making Identity Access Management (IAM) changes. Those changes are handled by a group of three administrators and are performed via an administrative account, only. This account is the only one able to make IAM changes. Later, we will also discuss how we control access to this account. By distributing credentials in this manner, we feel we have much better protection of our infrastructure in the cloud that, in turn, allows us to keep our customers data more secure. Managing Security Around Remote Login and Development With the earlier background on credential management, let s now look at how we manage security around remote login and remote development. Again, we will use AWS as the specific example in our discussion. Remote Server Login One major advantage in using the CloudBees PaaS is that you do not have to manage servers anymore. Using our platform, developers develop, deploy and scale applications with minimal server interaction. Behind the scenes, however, CloudBees engineers do need to manage server lifecycle. This is true not only for instances that run customer code, but for web proxying layers, databases, Git/SVN repos, and many other administrative areas. Earlier, we discussed the credentials that allow developers to see, and perhaps manage, the lifecycle of these servers. However, we also need to manage the ability to remotely login to these machines to perform maintenance or fix problems that may occur. In addition, we need to limit traffic from the outside world in a way that allows applications to work, but does not allow malicious attempts to break into the systems. Locked Down Access Our first strategy is to make prodigious use of EC2 security groups and rules. Each of our instances has a particular role it serves, and as such is tied to a specific security group that reflects that type of role. Our application servers, our proxying layer and our databases each have separate EC2 security groups attached to them. Within our development services, DEV@cloud, our Jenkins master instances, the executor machines and the proxying layer also have their own EC2 security groups. It is within these security groups that we can restrict outside traffic to only the ports needed and then also limit inside traffic between the EC2 security groups, where things need to "talk" internally. For example, our web proxying layer allows outside traffic from ports 80 and 443 - and that's it. Our application servers don't allow outside traffic at all, and only allow connections to specific ports coming from the web proxying layer. This tiered and locked down approach ensures we don't succumb to attackers looking for a backdoor into our environment. 6
Backdoors Of course, we still do need backdoors into the systems in order for our own team to get in and perform administrative tasks. Most commonly this access includes remote login (SSH) to a server, but also includes access to backend web interfaces to monitor application health or observe application metrics in order to solve issues. To ensure we maintain as much security around these backdoors as possible, we hide them all behind a Virtual Private Network (VPN) that is accessible only to CloudBees developers. We use openvpn, which is a userspace-based SSL VPN that tunnels traffic over UDP. Each developer who has the need for access is given a private key to access the VPN. Once established on the VPN, the developer now has access to the ports needed to get into the system. Note that this does not mean developers automatically have access into the systems; it just means they have access to the mechanisms to get into the systems. Case in point: once on the VPN, developers have access into port 22 (SSH) on our various machines. However, this still doesn't mean they have the access keys to actually login to those various systems - this is a separate credentialing and distribution mechanism that is handled on an as-needed basis. This two-layer approach gives us a high level of security, while still maintaining usability for our development team. Handling Problems While it provides security, the VPN system can still be a source of friction. Maintenance, or an unplanned outage on the VPN system itself, can halt developer progress across the entire system. In a way, the VPN becomes a single point of failure for our team to be able to handle system level issues, should they occur. To handle this problem, we allow our administrators to make temporary rule changes to the EC2 security groups. This capability facilitates work on system issues if the VPN system itself becomes a bottleneck to progress. As an example, they can open SSH access to a specific external IP address a developer may be using in order to let them login while bypassing the VPN. This change can only be facilitated by an administrator. In addition, our security group rules are monitored by an external script on a nightly basis. A script matches the state of the security group rules with a known state stored in a Git repository; any deviations are noted and an email is generated. This mechanism allows all administrators to keep tabs on rule changes and ensure "temporary" changes get reverted, or made permanent by adding them to the Git repository of "good" rules. We feel that our VPN approach, coupled with continuous auditing of security group rules against a known standard, provides us with a very high level of overall security around external facing access into our critical infrastructure. This, in turn, provides our customers with the highest levels of security against intrusion and potential data theft. 7
Credentials and Password Policies We will now examine some best practices in credentials and password policies used by CloudBees in managing our PaaS environment. These are important practices that are fairly easy to follow without causing undue burden on the development team. Centralized Password Management Aside from infrastructure cloud providers, like Amazon Web Services, we have a number of external service providers we use to run the day-to-day business at CloudBees. For many of these providers, we have a single account that is used by CloudBees to provide services that are both internally and externally facing. From time to time, our developers may need to log into these services to update information, check settings or get reports. As noted earlier, having a single set of credentials is an undesirable scenario. Each new developer has to be given those credentials and it's hard to keep track of exactly how far and wide they float around. In addition, if a developer leaves or some event happens that necessitates change, then redistributing the new credentials becomes painful. As a small team it was less of a problem, but as the team grew a better way of handling this had to be developed. For the past year and a half, we have tackled this problem largely through the provider we use, OneLogin (http://www.onelogin.com). OneLogin Each developer has a separate login to our OneLogin system. This can be username/password based, or can be synced with a Google Apps account to allow use of a Google Apps login via OpenID. Once logged in to OpenID, developers are presented with a menu of services that have been predefined for them to be able to log into. Links are provided directly to the various service providers page for automated login. Developers do not have to remember any specific login credentials, and account passwords don't have to be shared. OneLogin makes it easy for us to control access to all of the important CloudBees systems (including logging into the AWS console). An obvious downside is that OneLogin is now a single point of failure for access to our critical infrastructure. However, we have never experienced a technical issue that has prevented us from using the service. Our admins also maintain a privately secured username/password list of various logins, just in case. As an extra layer of security, our administrators use two-factor authentication via a cell phone app when signing into OneLogin. Password Resetting One often-overlooked area in security control is the email address used for service registration. For many of our services, we register an alias@cloudbees.com email address that gets forwarded to the development team using that service. This is great for informational updates, but it also means that password reset information goes to that alias -- and then on to the email address of the team members from whom we may be trying to abstract that information. 8
This is one important detail we have focused on in setting up our Amazon EC2 accounts. We want to ensure that if a set of credentials did get stolen, particularly from a hacked email account, that an attacker would find it difficult to reset an account password in order to gain access to a system. Keeping Credentials Private At CloudBees, we are big fans of GitHub, and we have a number of public and private repositories there. However, we also maintain a local private Git server for a handful of critical repositories that, for internal security reasons, we've decided are better kept closer to home. This set of repositories includes ones that don't need to be widespread and are more critical for behind-the-scenes operation. We tend to keep repositories that may have services credentials stored in them contained completely within our own infrastructure, as opposed to storing them in a third party location. Email As a matter of policy, we never use plain text email to send important credentials to end users. When sharing AWS credentials with a new developer, or getting an SSH key set up, we always transact using GPG-encrypted data. This added step adds only about 30 seconds to the process, but helps ensure if a developer's email is ever compromised, then credentials are not part of the leaked data. We have found that a lot of developers use email as a permanent archive of data. If you send them some kind of login information, it will stay in their inbox/archive forever for "future reference." By sending encrypted credentials, the credentials will continue to be available in the future, if needed, but will ensure that security breaches won't lead to data loss. Conclusions As a developer or IT operations person, you almost certainly have existing security practices and procedures in place. Platform as a Service changes the level at which you interact with underlying infrastructure resources, so it necessitates a reexamination of your existing practices. Operations that would previously be performed by a sysadmin are, within the PaaS, performed by services. The security of those services, and the way in which you and the PaaS provider operate them must be considered. If you are attempting to implement a private PaaS yourself, you will need to consider many of the solutions used by CloudBees already in our full service PaaS offering. In this paper, we have documented much of CloudBees own security practices as well as best practices overall, and the ways in which PaaS changes security considerations. For more information or a deeper dive into our security practices (under non-disclosure), please contact CloudBees directly: info@cloudbees.com. 2012 CloudBees, Inc. CloudBees is a registered trademark and DEV@cloud, RUN@cloud and AnyCloud are trademarks of CloudBees, Inc. Other product or brand names may be trademarks or registered trademarks of their respective holders. 060612v00 9