Why we Picked CF as the Basis for our Public Cloud Multi-Tenant Platform Mike Root @mikersj Jeroen van Rotterdam @jvanrotterdam 1
applications THE ANNOYING Our Stuff is Complex tenants WE ONLY WANT TO BUILD APPLICATIONS 2
THE ANNOYING Our Stuff is Complex Shared Application Micro Services WE ONLY WANT TO BUILD APPLICATIONS 3
TENANT PARTITIONING Acme Supplier Exchange Client acme.emcond.com FooBar Supplier Exchange Client FBPortal.emcond.com Router acme.emcond.com FBPortal.emcond.com : se v16 : se v15 Supplier Exchange v16 Supplier Exchange v15 v15 s Tenant Admin v1 Shared Services V7 Shared Services V7 Shared Services V6 Shared Services V6 xdb Server xdb Server Cassandra Server Cassandra Server SWIFT SWIFT SWIFT xdb cluster (metadata) Cassandra cluster Cloud Blob Store 4
PRODUCTION SETUP Warden Containers CloudFoundry BOSH managed VM s Concurrent Authoring Service Analytics Engine Transformation Services Metadata Service Platform Mgt Console Tenant Mgt Console Supplier Exchange BPM Service Case Management Service Retention Mgt Service Other VM s Windows VM AD/ADFS/vCenter etc HAWQ Pivotal HD Secure Full Text Engine Swift Blob Store ClamAV (virus check) Firestone Authentication CF Service Cassandra CF Service xdb CF Service RabbitMQ 5
NETWORK SEGREGATION BOSH CF Other CF Router DEA CF Services Windows 6
Example CLOUDFOUNDRY BENEFITS BOSH provisions the entire datacenter Repeatable at any time No downtime Cloud Foundry Handles application scalability Tenant (Customer) modifiable URL Upgrade tool Uses the CF API Blue Green upgrade (0 downtime) Dynamically move/configure tenants Promotion process is fully automated Shellshock, OS Security vulnerability 1 We updated the stem-cell (OS) 2 Ran bosh deploy 16 CF environments updated (dev/test/pre-prod/prod) 3 16x30 VM s updated No down time Drank beer (optional) 7
CLOUDFOUNDRY BENEFITS WE DIDN T BUILD ANY OF THIS Monitoring Monitoring VMs Monitoring applications Resource scaling High Availability VM HA Application HA Log collection Health metrics Shellshock, OS Security vulnerability WE ONLY WANT TO BUILD APPLICATIONS 8
SPIFF GENERATION OF BOSH MANIFESTS 16 Deployments managed with auto generated manifests Core templates 16 Instance specific deployments Support for multiple networks Contribute to the community $> spiff merge cf-jobs.yml cf-network.yml vcenter.yml ci.yml dev.yml preprod.yml prod.yml ci_merged.yml dev_merged.yml preprod_merged.yml prod_merged.yml 9
CLOUDFOUNDRY UPGRADE TOOL T0-A1 T0-A4 T1-A1 T2-A1 T3-A1 T5-A1 T4-A1 T3-A2 T0-A2 T0-A3 Tenant - App configuration BLUE GREEN DEPLOYMENT App Blue App Blue App Green NGIS Blue NGIS Green CloudFoundry CloudFoundry API to deploy applications SaaS REST to move/configure tenants e.g. new security rules, enable features etc Old and new versions running in parallel 10
GAPS WE CONQUERED OS hardening; Ubuntu patch version, ssh permissions, file access etc etc Deploying in multiple networks with firewalls CF/BOSH/DEA/Router/Services Spiff not setup for multiple networks Keeping up to date with latest CF release Converting v1 services to v2 services DR setup CF cli is constantly changing, but CF api is more stable We use the API for the upgrade tool Debugging applications in CloudFoundry (step through code) Filesystem options/type for persistent storage (swift) Filesystem size of /tmp is not configurable BOSH builds are more difficult because BOSH doesn t support artifactory as a BLOB store Managing multiple BOSH deployments Log files other than stderr/stdout Collecting them Limited disk space SPIFF manifests are harder to read by a human Spiff diff helps Application needs to report ready before ready for large application Router networking resources exhausted keep-alive disabled CF staging blobs not cleaned up properly CF routes not cleaned up properly WE ONLY WANT TO BUILD APPLICATIONS 11
CONTRIBUTIONS TO CLOUNDFOUNDRY Bosh releases ClamAV First EMC Contribution to CF.org SWIFT HA Proxy Deployment VM BOSH CLI CF CLI User management ability Cassandra as a CF Service Zabbix Agent (open source monitoring tool) Network yml Service Brokers Cassandra xdb WE ONLY WANT TO BUILD APPLICATIONS 12
PROMOTION PROCESS CI SaaS Continuous Deployment Process DEV Validation Build #806 Build #805 Build Build #803 Build #802 QE Validation Functional Test L10N/I18N Test 6 hours Performance Validation Performance Test Longevity Test 12 hours Integration Validation Integration Test Upgrade Validation 30 minutes Upgrade Test Pre-Prod Validation Environment Test 28 mins Production Datacenter 1 Datacenter 2 Datacenter 3 Datacenter 4 Cloud Foundry Cloud Foundry Cloud Foundry Cloud Foundry Cloud Foundry Cloud Foundry Cloud Foundry 1 2 3 4 5 6 7-8 Upgrade Tool / BOSH: Configuration Management, Orchestration 13
DEPLOYMENT HISTORY 8 BOSH upgrades 2 CloudFoundry upgrades 22 upgrades of 3 applications 1 stemcell upgrade 29 releases in 37 weeks since GA GA 14
Q & A NOT THAT WE HAVE TIME FOR THIS 15