Challenges and lessons learned with Openstack deployments and MySQL Sandro Mazzio+a Sr Director Product Management IaaS
Who I am o o o o o I am not a DB expert or a DBA I work in Product Management since 15+ years I work at enovance since 2 months I started with Openstack in July 2011 I ve been launching HP Cloud Services DBaaS offering
enovance loca>ons and figures Montréal Bangalore Founded in San Francisco Paris People Customers Growth in Revenue
Journey to the cloud Software Value Engineering Software Architecture : from centralized apps to distributed services Software Engineering : from Waterfall to Agile Client / Server Vertical Scaling Hardware is key Centralized Services / API Horizontal Scaling Software is key Distributed Infrastructure IT Infrastructure : from Mainframe to Cloud Expensive Hw Vertical Scaling Monolithic Centralized Cheap Hw Horizontal Scaling Distributed Cloud Infrastructure Engineering : from Ops to DevOps
Market & opportunity Large Enterprises Software for Private Cloud: $ 10 bn by 2017 Telcos xsp Software for Public Cloud : $ 8 bn by 2017 Not Scalable Expensive Not Cloud Not Scalable Expensive Not Cloud Rich Set of Features No Versioning Shared Infrastructure Containers Not Scalable Not Cloud enovance Cloud Solu/ons Seamless Build & Delivery of OpenStack Cloud Infrastructures Telecom Equipment Vendors NFV : $ 2bn by 2017 Migrate from Physical Appliance To Network Functions Virtualization
What we do? Build a Cloud Deliver your Private or Public Cloud on Premise with a real Cloud Approach : No Versioning Pay as you Grow AWS Features Match Enterprise Grade SLA 24/7 Support Use a Cloud Deliver Cloud-Ready applications on top of existing Cloud infrastructures : Elastic & Scalable Continuous Delivery Pay as you Grow Enterprise Grade SLA 24/7 Support
Openstack & enovance q Gold Member q Top Ten Contributor q Share DNA Open Cloud, Open Source
Openstack is not simple
Openstack is not cloud (Subbu Allamaraju ebay, July 25, 2013) infra onboarding, boostrapping, remediaron, config management, patching, packaging, upgrades, high availability, monitoring, metrics, user support, capacity forecasrng and management, billing or chargeback, reclamaron, security, firewalls, DNS, integraron with other internal infrastructure and tools, and on and on and on. All key requirements for producron: By Enterprise for Private Cloud By Service Providers for Public Cloud
Openstack Upstream does not address HA Openstack HA Guide h+p://docs.openstack.org/high- availability- guide/content/ index.html ApplicaRon on top Openstack OS Infra. Plane: Openstack API Message Queues DB VirtualizaRon Layer
enovance Openstack Reference Architecture V1.0 Ø Management cluster full highly available Ø OpenStack API, schedulers Ø RabbitMQ, Database, Cache Ø Network services Ø Compute nodes with KVM and Open- vswitch with GRE tunnels as overlay Ø Swia cluster Ø Ceph cluster, used as backend for Nova, Glance and Cinder
enovance Openstack product Contains all core projects of Openstack Supports HA in a Single DC mode Based on edeploy (enovance Bare Metal Deployment Opensource) Puppet as Soaware ConfiguraRon Management
What about DB in our product? DB Requirements: FuncRonal & Performance requirements: Galera cluster for MySQL 100 % uprme Performance DegradaRon is ok Business Requirements: Opensource
enovance Galera for MySQL usage: results Experience from ProducRon Environment: 1 cluster of 3 Management Nodes No Service InterrupRon Performance DegradaRon are OK Galera performed correctly
On the topic of Mul> Data Center East ASIA Goal is to address: West 1 EMEA Disaster Recovery Local Affinity
Not all services are ready for Mul>- DC Openstack has a concept of AZ, Host AggregaRon = good in a single DC. Not good If your DC are separated by a WAN Nova Cells is good for distriburon (but nobody else is taken care) MulRple Region Keystone in every region or one overlay Using Heat to orchestrate MulRple Region Galera 3 Galera SynchronizaRon over WAN
Keystone Our experience working at large customers is that: Token storage is a nightmare and and do not scale well on large scale deployment Key Value Store (memcache, redis, ) - > MySQL - > memcache (but we have a bug) euh
Everything is MySQL, almost VS
Ceilometer, we use mongodb Because we need scalibility and because of the nature of informaron to store But Requires MongoDB DBA, Runbook, skills,
Openstack is not cloud (Subbu Allamaraju ebay, July 25, 2013) infra onboarding, boostrapping, remediaron, config management, patching, packaging, upgrades, high availability, monitoring, metrics, user support, capacity forecasrng and management, billing or chargeback, reclamaron, security, firewalls, DNS, integraron with other internal infrastructure and tools, and on and on and on. All key requirements for producron: By Enterprise for Private Cloud By Service Providers for Public Cloud
Why we need update upgrade patching? Billing CRM Portal seongs changes IntegraRon mess User new release bug fixes driver firmware update
What is our vision Deliver Openstack and app on top of it using Openstack process and tools ConRnuous IntegraRon/Delivery
Leverage the Openstack process How Openstack handle Contribu/on of 400 developers / month? Developers from 250 companies? 10+ integrated projects? and yet release a new version every 6 months with 400 new features without major inconsistencies?
and tools Everything in Process descrip/on and code Infrastructure used to handle the process Code to deploy the infrastructure 100 Rmes be+er than documentaron of a manual process that is never up to date... Reproducible environments Dynamic Provision test env when needed by Jenkins Allow dev to generate a test env for debug at will Store validated releases and customer specific env (in Openstack Swi^)
Benefits Improve Efficiency Nothing is local, everything is shared Control access rights Contribute model No need for a full release team Factorize hardware needs Reproducible Env Dynamically provision test environments when needed by Jenkins Allow developers to generate a test environment for debugging at will Store validated releases and customer specific environments (in Swi^)
What we are describing is Devops Validate code changes and mulrple deployments environments at once Accommodate more devs working on the same projects Limit wait Rmes Industrializes development and...
Conclusion Openstack is not easy. It is ok at small scale in a single DC mode. enovance has a product to address Single DC HA mode deployment is applying Openstack CD/CI to Openstack deployment itself is working on the MulR DC mode
Ques>ons? Sandro Mazzioaa sandro.mazzioaa@enovance.com @smazzioaa