Openshift for Continuous Integration Alex Lossent IT/PES/IS AI meeting 1-Oct-2015 Openshift for Continuous Integration 1
Outline Continuous Integration: context and history Platform-as-a-Service concept Docker/Kubernetes/Openshift How Openshift compares to Puppet, Heat Plans and outlook AI meeting 1-Oct-2015 Openshift for Continuous Integration 2
Why a Continuous Integration platform? Concept: improve code quality by integrating code frequently and applying automatic quality control (tests etc.) Awareness of multiple CI setups for all large software projects at CERN In IT, PH, GS, EN Using Bamboo, Jenkins, TeamCity First experience with Jenkins in PES: AIQA Project to provide a Jenkins platform as a service to make it easier for developers to deploy CI AI meeting 1-Oct-2015 Openshift for Continuous Integration 3
CI: current offer Since end 2014, we provide a CI platform Using well-established, open-source Jenkins A Java web application Request an instance from http://cern.ch/forge Documentation http://cern.ch/jenkinsdoc Clear interest from developers >40 instances as of September 2015 Developers@CERN forum on 28-29 September A platform rather than a service We provide the infrastructure (Puppet/Openstack) Out of scope: support for Jenkins configuration itself, integration of existing CI setups Full details presented at ITTF 5-Dec-2014 AI meeting 1-Oct-2015 Openshift for Continuous Integration 4
Significant challenges Need more flexibility Every project has specific needs No standard development platform at CERN Frequent requests for custom RPMs, libraries, OSes, plugins, integration scenarios etc. Need more automation Many changes trigger a support ticket Too much manual work for both users and ourselves Need better scaling 1..n VM per project with CI Security Team project to provide static code analysis AI meeting 1-Oct-2015 Openshift for Continuous Integration 5
Towards a PaaS solution We want to provide the infrastructure for CI While giving maximum freedom to Jenkins users With minimum cost of entry and maintenance effort for users I.e. self-service, fully automated, self-repairing A good candidate for a PaaS solution AI meeting 1-Oct-2015 Openshift for Continuous Integration 6
Options to deploy a web app SaaS: Drupal SharePoint Less effort Less flexibility Specific help Web hosting: AFS/DFS website Middleware-ondemand to develop, run, and manage Web applications without the complexity of building and maintaining the infrastructure typically associated with developing and launching an app [Wikipedia] PaaS: Good flexibility Limited effort IaaS: Openstack Puppet DNSLB More effort Best flexibility Unspecific help AI meeting 1-Oct-2015 Openshift for Continuous Integration 7
PaaS: useful technologies Containers Operating System-level virtualization: A host system runs several isolated user spaces on top of a single kernel Provides app isolation with less overhead than VMs Containers can be provisioned and started in seconds Allow to separate maintenance of the host OS from that of the application itself Like VMs can stop worrying about physical hardware Docker A popular container solution An application can be packaged with all its environment into an image Includes all libraries, app framework, code, configuration Portable: can be deployed and instantiated with little constraints on a variety of hosts AI meeting 1-Oct-2015 Openshift for Continuous Integration 8
PaaS: useful technologies Kubernetes Docker container orchestration system by Google Software-defined network (SDN) to provide advanced services Recent but gets a lot of traction RedHat Openshift Builds a PaaS solution on top of Kubernetes Adds user management, UI, web proxy, image registry, application templates etc. And an application lifecycle (Continuous Deployment) AI meeting 1-Oct-2015 Openshift for Continuous Integration 9
What can we expect from using Openshift? Flexibility Delegate full control on the Jenkins app to users Self-service and automation (by design) Scaling Better resource-efficiency with containers Advanced level of service (from Kubernetes) Persistent storage via Ceph (RBD) Fast recovery (virtual IPs, replication controller) Good availability (container migration) AI meeting 1-Oct-2015 Openshift for Continuous Integration 10
How to apply this to CI? We provide and maintain a reference Jenkins Docker image With relevant pre-configuration and integration And an Openshift application template for it Users create instances of Jenkins on our Openshift cluster (self-service via CLI/GUI) Allow user-provided Docker images for slaves Each project can have its own build environment No need to deploy and maintain dedicated VMs Fresh container for each job run AI meeting 1-Oct-2015 Openshift for Continuous Integration 11
PaaS general use case Openshift interesting for general web application hosting ~1400 HTTP-related openings in the external firewall Applications currently hosted outside CERN More flexible than web hosting/mwod Wide choice of frameworks, languages Significantly less effort than IaaS No host OS to manage Configuration management tailored for web apps Scalability, redundancy, good availability are almost free Fills a gap in providing tools to facilitate web app deployment and upgrade AI meeting 1-Oct-2015 Openshift for Continuous Integration 12
Case study: GitLab IaaS-based deployment Many VMs (dev/prod, redundancy with DNSLB) NFS filer, DBoD (single points of failure) Complex configuration management with Puppet Puppet module + tricks, custom Lemon sensors Significant work with each new version Complex, manual upgrade procedure 30 minute downtime (even for possibly online upgrades) Not doable with Puppet (lack of synchronization), even less so with migration of Puppet repos to GitLab Slow recovery in case of failure DNS update delay Can be much longer if DBoD or NFS filer fail (if possible) AI meeting 1-Oct-2015 Openshift for Continuous Integration 13
Case study: what Openshift could improve Better resource efficiency: a lot less VMs Direct use of Ceph (RBD) volume for storage Single GitLab instance ( pod ) at any given time Faster recovery Takes seconds to reschedule a container, reconnect to storage, reassign virtual IP SPoF (and dependency loop) removal: NFS filer possibility to host database in Openshift as well Considerably simplified management and procedures Apps are described in terms of how to generate a Docker image from source code and a deployment configuration Openshift designed to facilitate Continuous Delivery Automation of building Docker images, deploying new app versions (and rollback) Can leverage standard RoR migration/upgrade procedure AI meeting 1-Oct-2015 Openshift for Continuous Integration 14
Caveat: security updates App owners responsible for maintaining their apps and all their dependencies Security fixes! Openshift provides an (optional) workflow whereby a base image can be updated and app rebuilt on top of it Mostly appropriate for apps with a Continuous Deployment model Frequent integration, updates and redeployment AI meeting 1-Oct-2015 Openshift for Continuous Integration 15
An interesting complement to existing web app hosting offer SaaS: Drupal SharePoint Less effort Less flexibility Specific help Web hosting: AFS/DFS website Middleware-ondemand IaaS: Openstack Puppet DNSLB More effort Best flexibility Unspecific help PaaS: Good flexibility Limited effort AI meeting 1-Oct-2015 Openshift for Continuous Integration 16
What about Heat? Openstack Heat was also considered as a way to address current challenges with CI Heat = orchestration for Openstack Can be used to provide a Jenkins application template Improvements regarding automation, self-service, flexibility This option would have been fantastic if available end 2014 when CI project started But now suffers from availability of a solution like Openshift Non-trivial work required: integration of Heat with Puppet to maintain long-lived VMs complete refactor of how Jenkins is managed by Puppet Resource efficiency level of service (recovery etc.) AI meeting 1-Oct-2015 Openshift for Continuous Integration 17
Where are we now? Prototype Openshift cluster was relatively easy to deploy Hosts managed with Puppet (+Ansible) Polishing to do but no blocker in sight Work on a preconfigured Jenkins image in progress With SSO, GitLab integration, etc. Production status before end of the year is still realistic AI meeting 1-Oct-2015 Openshift for Continuous Integration 18
Collaboration opportunities? Docker has been lurking around for some time, but no well-defined strategy yet In particular, no official SLC5/6 or CC7 images A new type of resource to manage Reassign when users leave CERN, etc. Integration with existing services? New technologies to master Docker, Kubernetes are definitely technologies that will play a major role in the coming years AI meeting 1-Oct-2015 Openshift for Continuous Integration 19
Do you want to know more? Openshift 3 Roadshow: free workshop in Geneva on October 22nd AI meeting 1-Oct-2015 Openshift for Continuous Integration 20