Deep Dive: Infrastructure as Code

Deep Dive: Infrastructure as Code Steven Bryen Solutions Architect, AWS Raj Wilkhu Principal Engineer, JUST EAT Bruce Jackson CTO, Myriad Group AG 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

You are on-board Business Continuous Delivery DevOps Cloud needs to experiment, innovate, reduce risk of services and applications culture, automation, measurement, sharing infrastructure-as-code

AWS CloudFormation Create templates of the infrastructure and applications you want to run on AWS Have the CloudFormation service automatically provision the required AWS resources and their relationships from the templates Easily version control, replicate or update the infrastructure and applications using the templates Integrates with other development, CI/CD, and management tools.

Create template Example web application CloudWatch alarms Software pkgs, config, & data DB Service Inventory Service Elastic Load Balancing EC2 instance Recommendations Service Auto Scaling group security group ElastiCache memcached cluster

Create template Resources CloudFormation Template CloudWatch alarms Elastic Load Balancing Software pkgs, config, & data EC2 instance Auto Scaling group security group ElastiCache memcached cluster "Resources" : { "SecurityGroup" : {}, "WebServerGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "MinSize" : "1", "MaxSize" : "3", "LoadBalancerNames" : [ { "Ref" : "LoadBalancer" } ],... } }, "LoadBalancer" : {}, "CacheCluster" : {}, "Alarm" : {} },

Create template Parameters EC2 instance Auto Scaling group Recommendation Inventory Service s Service Customer DB Service Info to Customize Stack at Creation. Examples: Instance Type, App Pkg Version CloudFormation Template "Parameters" : { "CustomerDBServiceEndPoint" : { "Description" : "URL of the Customer DB Service", "Type" : "String" }, "CustomerDBServiceKey" : { "Description" : "API key for the Customer DB Service", "Type" : "String", "NoEcho" : "true" }, "InstanceType" : { "Description" : "WebServer EC2 instance type", "Type" : "String", "Default" : "m3.medium", "AllowedValues" : ["m3.medium","m3.large","m3.xlarge"], "ConstraintDescription" : "Must be a valid instance type"

Create template Outputs CloudFormation Template Elastic Load Balancing "Resources" : { "LoadBalancer" : {},... }, "Outputs" : { "WebsiteDNSName" : { "Description" : "The DNS name of the website", "Value" : { "Fn::GetAtt" : [ "LoadBalancer", "DNSName" ] } } }

Create template Deploy and configure software Software pkgs, config, & data CloudFormation Template "AWS::CloudFormation::Init": { "webapp- config": { "packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {}, "commands" : {}, "services" : {} }, ü Declarative ü Debug-able ü Updatable ü Highly Secure ü BIOT Bring In Other Tools } "chef- config" : {} EC2 instance Auto Scaling group

Create template Language features

Use a wide range of AWS services ü Auto Scaling ü Amazon CloudFront ü AWS CloudTrail ü AWS CloudWatch ü Amazon DynamoDB ü Amazon EC2 ü Amazon ElastiCache ü AWS Elastic Beanstalk ü AWS Elastic Load Balancing ü Amazon Kinesis ü AWS Identity and Access Mgmt ü AWS OpsWorks ü Amazon RDS ü Amazon Redshift ü Amazon Route 53 ü Amazon S3 ü Amazon SimpleDB ü Amazon SNS ü Amazon SQS ü Amazon VPC As of April 2015 and more

Nested CloudFormation Stacks Pass parameters to nested CloudFormation Stacks "mystackwithparams" : { "Type" : "AWS::CloudFormation::Stack", } "Properties" : { } "TemplateURL" : "https://s3.amazonaws.com/template-bucket/mystack.template", Parameters" : { "InstanceType" : "t1.micro", } "KeyName" : "mykey"

Iterate on infrastructure

Update stack In-place Blue-Green Faster Cost-efficient Simpler state and data migration Working stack not touched

Extending AWS CloudFormation

Extend with Custom Resources CloudWatch alarms Software pkgs, config, & data Provision AWS Resources AWS CloudFormation Create, Update, Rollback, or Delete + Metadata Success + Metadata Web Analytics Service Elastic Load Balancing EC2 instance "Resources" : { "WebAnalyticsTrackingID" : { "Type" : "Custom::WebAnalyticsService::TrackingID", "Properties" : { "ServiceToken" : "arn:aws:sns:...", Auto Scaling group security group ElastiCache memcached cluster },... } "Target" : {"Fn::GetAtt" : ["LoadBalancer", "DNSName"]}, "Plan" : "Gold"

Lambda-powered custom resources CloudWatch alarms Software pkgs, config, & data // Implement Custom Logic Here Elastic Load Balancing EC2 instance Lookup an AMI ID Lookup VPC ID and Subnet ID Auto Scaling group security group ElastiCache memcached cluster Reverse an IP Address

Application Lifecycle

Infrastructure Lifecycle Application Lifecycle CloudFormation Templatize Replicate Automate EC2 SQS, SNS, Kinesis, etc. Databases VPC IAM Download Packages, Install Software, Configure Apps, Bootstrap Apps, Update Software, Restart Apps, etc.

AWS::CloudFormation::Init Declarative "AWS::CloudFormation::Init": { "webapp-config": { "packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {}, "commands" : {}, "services" : {}

AWS::CloudFormation::Init Supports updates "packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {}, "commands" : {}, "services" : {}

AWS::CloudFormation::Init Flexibility to bring in other tools such as AWS CodeDeploy and Chef "install_chef" : {}, "install_wordpress" : { "commands" : { "01_get_cookbook" : {},..., "05_configure_node_run_list" : { "command" : "knife node run_list add -z `knife node list -z` recipe[wordpress]", "cwd" : "/var/chef/chef-repo", ow.ly/dinkz "env" : { "HOME" : "/var/chef" }

Use AWS::CloudFormation::Init "UserData": { "# Get the latest CloudFormation helper scripts package\n", "yum update -y aws-cfn-bootstrap\n", "# Trigger CloudFormation::Init configuration \n", "/opt/aws/bin/cfn-init --stack ", {"Ref": "AWS::StackId"}, " --resource WebServerInstance ", " --region ", {"Ref": "AWS::Region"}, "\n", "# Signal completion\n", "/opt/aws/bin/cfn-signal e $? --stack ", {"Ref": "AWS::StackId"}, " --resource WebServerInstance ", " --region ", {"Ref": "AWS::Region"}, "\n"

Use CloudWatch Logs for debugging "install_logs": { "packages" : {... "awslogs"... }, "services" : {... "awslogs"... } "files": { "/tmp/cwlogs/cfn-logs.conf": {} file = /var/log/cfn-init.log log_stream_name = {instance_id}/cfn-init.log file = /var/log/cfn-hup.log log_stream_name = {instance_id}/cfn-hup.log ow.ly/e0zo3

Use CloudWatch Logs for debugging ow.ly/e0zo3

Wait Conditions & DependsOn

Example Wait for EC2 instance to signal success "Ec2Instance" : { "Type" : "AWS::EC2::Instance", }, "Properties" : { "UserData" : { "Fn::Base64" : {"Ref" : "mywaithandle"}}, "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]} }

Example Wait for EC2 instance to signal success "mywaithandle" : { "Type" : "AWS::CloudFormation::WaitConditionHandle", }, "mywaitcondition" : { } "Type" : "AWS::CloudFormation::WaitCondition", "DependsOn" : "Ec2Instance", "Properties" : { } "Handle" : { "Ref" : "mywaithandle" }, "Timeout" : "4500"

CI/CD Pipeline

CloudFormation in a CI/CD Pipeline App Developers App Pkgs, CloudFormation Templates, Etc. App Code & Templates Issue Tracker Dev Env Code Repo AWS CI Server CloudFormation Test Staging "Infra- as- Code" DevOps Engineers, Infrastructure Developers, Systems Engineers Code Review Prod

LONDON

Continuous Integration and Deployment Raj Wilkhu Principal Engineer, JUST EAT 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

JUST EAT Platform High volume e-commerce platform 45-60 minutes cycle Real time message delivery and confirmation network Hardware in 22,500 restaurants in the UK, 40,800 worldwide Peak traffic is 10,000% of normal daytime traffic 3 cities Over 130 releases a month Primarily Windows

Building reliable and high quality software Test-Driven Development, Continuous Deployment and Immutable Infrastructure Every part of the infrastructure replaced several times a day Incentive to continuously speed up deployment process

Deploying components Platform decomposed into components Each component defined by single AWS Cloudformation template Single Build Artifact Instances boot and bootstrap themselves from pre-baked AMI Content in S3 based on Cloudformation::Init metadata

Pre-baked AMI vs Dynamic config Pre-baked AMI CloudFormation::Init Increasing Boot time System dependencies (eg OS features,.net, ruby, gems, etc) Static configurations Baked via CI pipeline Tested and tagged Continuous Deployment (latest code) Environment specific configuration Automation to deploy feature Unzip package Install and configure dependencies Execute deployment script(s) Warm up feature Tagged by deployment scripts

Cloudformation template generation Master Single json from multiple yaml and erb files Similar to HTML templating engines Organized by component Convention over configuration { Example yaml: --- :feature_config: :ami_id: ami-c99544ef :elb: true :use_logging_client: true :elb_dns: true :parameters: :ElbTarget: "HTTP:80/status" :AsgDesiredSizeDuringPeakTime: 10 :InstanceType: c4.xlarge Example json.erb template: "<%= @feature_name %>_pre_deploy": { "commands": { <% cmd_prefix = 250 services_to_disable = %w(aelookupsvc bits scdeviceenum trustedinstaller wuauserv) services_to_disable.each do service_name %> "<%= cmd_prefix %>_<%= @feature_name %>_stop_<%= service_name %>_service_because_unnecessary": { "waitaftercompletion": "0", "command": "powershell -noninteractive -noprofile -command \"& { stop-service - Name <%= service_name %> -verbose }\"",

Immutable infrastructure - Cloudformation Artifact JustDeploy Continuous Integration Server (TeamCity Agent Cloud) Artifact Rooms Tests in QA environment Build & Test Pipeline configs in yaml, template extensions in json.erb + master Component code Git Git Route 53 Engineers Engineers EC2 Web/worker roles Production/QA

Immutable infrastructure - Baking AMIs Continuous Integration Server (TeamCity Agenet Cloud) AMI Bakery Latest Amazon OS image configurations and resources Git EC2 base image Shared AMI Automated Tests Engineers config.yaml updated

Future Automatically select last tested AMI image Use lambda to trigger AMI builds based on AMI release notifications via SNS Failover to secondary region Publish more tools as OSS https://github.com/justeat

Yes, we re recruiting too. http://tech.just-eat.com/jobs @justeat_tech + @rajwilkhu http://tech.just-eat.com

Deployment on the fly Bruce Jackson CTO, Myriad Group AG 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

From IMPS to msngr database app server web server web server web server... x16 Move from a mid-2000 carrier platform to web scale with 38M users on 16 stacks in 6 months with a team of 4

Basic foundations of our approach Rules OSGi used for all application stacks Make use of as much managed infrastructure as possible Avoid any environment configuration Make deployment as familiar as coding Rationale Rapid creation of services against interfaces We don t have any devops so outsource management where possible No files/scripts that only one person understands The team is made up of developers

What did we do? Created utility libraries that use code annotations and AWS tags for configuration Use Eclipse features to define bundles to deploy on systems

What did we do (2)? Created Eclipse tooling to manage deployment & code/ feature checkout

A quick demonstration

LONDON