DevOps on AWS: Best Practices for Enterprise IT Teams

Transcription

1 : Best Practices for Enterprise IT Teams logicworks.net FOR LOGIC 2015 Logicworks

2 Executive Summary Amazon Web Services is a robust, powerful, and constantly evolving set of tools and resources but it takes a DevOps team to turn these resources into highly available, responsive, secure infrastructure. A well-executed DevOps approach enables development teams to concentrate on continuous improvement of their software applications, not on deployment and infrastructure updates, and empowers IT teams to become drivers of organizational innovation and agility. What is DevOps? The benefits of DevOps are widely known: seventy percent (70%) of senior IT leaders believe they need to implement a DevOps approach. At their most foundational level, DevOps practices tightly integrate applications with the infrastructure on which they run. Therefore, DevOps teams are often composed of application 2015 Logicworks [email protected] logicworks.net FOR LOGIC 2

3 developers and system engineers who have developed integrated deployment and provisioning processes. While parts of DevOps philosophy can be implemented in a traditional IT environment, the new possibilities of Infrastructureas-Code are the foundation for DevOps best practices. In traditional IT environments, provisioning new servers is an expensive, time-consuming, and often highly manual process that is opaque to business leaders and development teams. Cloud platforms like AWS empower teams to automate deployments with custom scripts and configuration management tools, allowing for a rapid provisioning and deployment schedule. The pressure to deliver software applications quickly continues to increase. This is part of why deployment automation has always been a crucial component of DevOps philosophy. But fully automated environments require a sophisticated development process, and in the rush to production, that can add unwanted complexity. This is why organizations need to be strategic about selecting the aspects of DevOps the team can currently handle. DevOps is a spectrum, and it is not true that one needs to implement every 2015 Logicworks [email protected] logicworks.net FOR LOGIC 3

4 DevOps best practice in order to enjoy some benefit from DevOps principles. DevOps is a spectrum, from most basic to advanced: Base Images "Golden Master" Automed Network Architecture Deployment Automation Auto Scaling NoOps ArchOps DevOps One can also think of this as a spectrum from ArchOps to DevOps. ArchOps is about laying the foundation the minimum viable product of the environment and DevOps is about building the automation. In many ways this is a complete paradigm shift for the governing principles of most IT departments: there was a time when a system administrator would show the uptime on his systems with pride. DevOps teams take equal pride in high availability. This evolution should be considered a new state of the art, as IT shifts from system to application management Logicworks [email protected] logicworks.net FOR LOGIC 4

5 ArchOps: Laying the Foundation ArchOps is the set of principles that focuses on automating just the infrastructure buildout to be able to deploy standard environments in a few hours or days. At the most basic level, ArchOps is creating a perfect image of an instance to bring up manually. A good base image should not make any assumptions about what its name is, should be destructively tested, etc. An automated network architecture buildout is the next level of complexity. Even if an enterprise has hundreds of complex apps, the network configuration often does not change radically, and remembering to close specific security loopholes each time a network is configured has a high risk of failure. Setting up credentials and baking in naming conventions will also save large amounts of time. DevOps teams often use CloudFormation, a native AWS resource, to do this. If the network infrastructure ever needs to change, changes can be made in the CloudFormation template. This helps prevent regressions in a security policy and acts as a physical document that fully defines the environment being built. CloudFormation allows teams to automate things like deploying secure, multi-az web servers, network infrastructure, and can even download Puppet scripts and configures the Puppetmaster 2015 Logicworks [email protected] logicworks.net FOR LOGIC 5

6 dozens of small tasks that are very time-consuming if done manually. CloudFormation is especially time-saving if an enterprise s standard deployment has HIPAA or PCI compliance restraints or must incorporate legacy systems; a team can spend countless hours writing and re-installing 3rd party tools and a custom mix of AWS tools for each deployment. While this might be sustainable if with a small deployment of a handful of instances, one would need a fleet of NOC engineers to spin up hundreds of environments in an enterprise or growing start-up. In essence, CloudFormation allows teams to maintain buildout velocity by orchestrating AWS resources to spin up the foundation of an environment quickly. This is the fundamental goal of ArchOps. AWS DevOps Practices Once CloudFormation does the work of installing and configuring Puppet, now it is time to start pulling the strings. At this point, an IT team is getting into true DevOps practices. Configuration Management While it is possible to deploy a highly available Amazon Web 2015 Logicworks [email protected] logicworks.net FOR LOGIC 6

7 Services (AWS) environments without configuration management tools like Puppet, doing so significantly increases the risk of error and downtime. On any kind of infrastructure, automated software installs can save a team a great deal of time, but in a dynamic cloud environment, configuration management tools can actually create the AWS resources nodes require to operate, such as Elastic IPs, network interfaces, or block storage. A DevOps team relies on configuration management to maintain a single source of consistent, documented system configuration. As enterprise infrastructure becomes code and instances can be spun up or down with a few clicks, the protection this affords is absolutely essential for complex deployments. An EC2-resident configuration tool like Puppet is the key configuration engine during auto scaling events, provides version control, and has monitoring and reporting capacities, along with other benefits. It is often tempting to not bother with Puppet or configuration management in an initial AWS set-up, especially if the team is new to using Puppet. The benefit of doing the hard work upfront is that every consumer-facing application changes. The more that is automated, the more time engineers can spend on new projects and the quicker the team can adapt to change. Deployment Automation Deployment automation enables continuous integration, where 2015 Logicworks [email protected] logicworks.net FOR LOGIC 7

8 development teams deploy code to production several or even hundreds of times a day. With every commit that is pushed out to the site, a whole suite of unit tests verifies what is working. If a team has a one-button process to push from Git so they can deploy a large update to just one instance and then run statistics on that one node to make sure it is working correctly this is a very mature systems development process. AWS CodeDeploy is often used to facilitate this process. It has a number of functions, like deploying an update to a fleet of AWS EC2 instances (from a single instance to thousands) and automatically scheduling updates across multiple Availability Zones to maintain High Availability during deployment. Auto Scaling Auto scaling, when correctly implemented, significantly decreases the risks associated with traffic overflow causing server failure. At the same time costs are reduced; instead of running instances based on projected (assumed) usage and leaving excess resources in place as a buffer, one can only run resources matched to actual usage, on a moment-to-moment basis. These price and scalability advantages are not without their own complexities. Most importantly, auto scaling is a capacity of AWS, 2015 Logicworks [email protected] logicworks.net FOR LOGIC 8

9 but not a built-in function; it requires custom scripts and processes. There are many ways to getting a system from zero to 100% scalability, and configuration management tools like Puppet are the most common tools. A final step is integration between deploy process and auto scaling where Puppet scripts automatically integrate those EC2 instances into Auto Scaling groups. Puppet scripts can replace the need for a perfectly baked AMI. Instead, a vanilla template with the minimum possible configurations replaces individual Golden Masters for each server role. In this scenario, instance userdata or boot script needs only to do what is necessary to connect to the Puppetmaster. It takes a significant amount of time and focus to create and maintain these automation resources. Ultimately, a DevOps team may develop a sophisticated library of scripts that enable them to deploy testing environments or increase capacity within hours. These scripts themselves become a whole separate infrastructure that needs to be maintained. But the team is ultimately saving hundreds of hours of manual work and keeping development velocity high so it can spend time on code, not deploy. Choosing a Cloud Platform and Team 2015 Logicworks [email protected] logicworks.net FOR LOGIC 9

10 Start-ups and enterprises that are first moving into the cloud make one of two common mistakes: They either choose an out-of-thebox cloud solution that fits their immediate needs, or they hire a short-term consultant to help them migrate to the public cloud. Both solutions make it very difficult to implement a true DevOps culture in an IT team. Choosing a cloud platform with limited resources (that do not change frequently) often does not provide the breadth of automation and integration tools that keep DevOps teams innovating. And while a short-term cloud consultancy often promises to coach teams in DevOps philosophy, the team frequently struggles to maintain the infrastructure and does not have the time to innovate and explore once the consultants leave. This results in multiple, expensive engagements without affecting true internal change. Enterprises need to find a cloud platform that provides a sophisticated set of tools and constantly improves its infrastructure. They need a cloud platform with a broad development community that has used these tools to develop great infrastructure. They need templated deployment infrastructure tools. And importantly, most enterprises will require an a la carte resource model, so that these tools can be used in custom configurations. This is crucial to enterprises with complex builds that do not perform optimally in one-size-fits-all 2015 Logicworks [email protected] logicworks.net FOR LOGIC 10

11 systems. Currently, the only cloud provider that fits this description is Amazon Web Services. There is good reason why more enterprises choose AWS than any other IaaS provider. However, AWS is only as powerful as the team managing it longterm. Both the internal IT team and any Managed Service Provider the enterprise might bring on must understand the unique blend of AWS resources that put DevOps principles into action. Therefore, the combination of an experienced Managed Service Provider and AWS is much more powerful than a short-term consultant partner or Managed Services on top of a less robust cloud platform. Summary Wherever a team is on the ArchOps/DevOps spectrum, any degree of automation can lead to a gain in efficiency that directly translates into cost-efficiencies. A team that is less hampered by provisioning timelines, lengthy deployment approval processes, etc. will be able to concentrate on improving the organization s applications, not on repetitive manual work. IT will no longer be an organizational cost, but a driver of innovation, powering new lies of business and working more closely with business leaders to meet organizational goals. Although some teams may resist this 2015 Logicworks [email protected] logicworks.net FOR LOGIC 11

12 dramatic organizational shift, the inexorable progression of the industry will ultimately yield leaner, more agile teams than were possible in IT just five years ago. The world s largest enterprises now host complex, mission-critical infrastructure on IaaS platforms like Amazon Web Services. But the true power of AWS is only realized when the infrastructure is managed by a highly integrated DevOps team that is committed to process and communication. About Us Logicworks provides cloud computing and managed hosting to some of the world s most respected brands including Dow Jones, NBC, Lincoln Center, and Orion Health. As a Premier AWS Consulting Partner, Logicworks specializes in Hybrid and Managed AWS Solutions for SaaS, Media, ecommerce, Mobile, Healthcare, and Financial services companies. Visit us at to learn more or contact us at (866) FOR LOGIC Logicworks [email protected] logicworks.net FOR LOGIC 12