A central continuous integration platform Agile Infrastructure use case and future plans Dec 5th, 2014
1/3 The Agile Infrastructure Use Case By Stefanos Georgiou
What? Development practice Build better quality software, faster Developers integrate their work frequently Applying quality control with each integration Verified by an automated process Tests tests tests 12/5/2014 ITTF on Continuous Integration 4
You mean just testing compilation? Not quite.. Compiling Static & Dynamic Testing Unit & Integration Testing Code reviewing Deploying Documenting 12/5/2014 ITTF on Continuous Integration 5
Why? Postmortem bug-fixing is expensive Automate repetitive manual processes Maintain cohesive software and quality code base Visibility aggregated test data are showing trends No undeployable software It works on my machine 12/5/2014 ITTF on Continuous Integration 6
How? 12/5/2014 ITTF on Continuous Integration 7
Who? Top CI server in use Over 1000 Plugins Check out from your VCS automatically, run your tests, code coverage or static analysis, notify (e-mail, rss, jabber) 12/5/2014 ITTF on Continuous Integration 8
What's happening in Configuration Management? Our own jenkins master available at servant.cern.ch SSO & egroup-based job ownership support Daily building VMs and testing puppet modules CRM workflow runs here Not exactly pure CI QA process per change, multiple repos Getting there 12/5/2014 ITTF on Continuous Integration 9
12/5/2014 ITTF on Continuous Integration 10
CRM ticket process Shared module needs a change People using it probably come from different projects You can't just merge code to master Create CRM ticket in JIRA to notify for the change and wait 12/5/2014 ITTF on Continuous Integration 11
Previously in the CRM Workflow Write your puppet code inside a feature branch Test it yourself Create a ticket on the CRM JIRA project Merge to QA Hold for a week (or two) Any complaints? Merge to master 12/5/2014 ITTF on Continuous Integration 12
Rather do it like this Develop inside a feature branch 12/5/2014 ITTF on Continuous Integration 13
Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project 12/5/2014 ITTF on Continuous Integration 14
Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project Tick the box 12/5/2014 ITTF on Continuous Integration 15
Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project Tick the box Jenkins will run tests from your feature branch and merge it to QA for you 12/5/2014 ITTF on Continuous Integration 16
Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project Tick the box Jenkins will run tests from your feature branch and merge it to QA for you Hold for a week (or two) 12/5/2014 ITTF on Continuous Integration 17
Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project Tick the box Jenkins will run tests from your feature branch and merge it to QA for you Hold for a week (or two) Broken? 12/5/2014 ITTF on Continuous Integration 18
Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project Tick the box Jenkins will run tests from your feature branch and merge it to QA for you Hold for a week (or two) Broken? Jenkins will merge to master on proposed date after final test, assuming no problems found 12/5/2014 ITTF on Continuous Integration 19
Feedback from the build 12/5/2014 ITTF on Continuous Integration 20
What's being tested right now? Most modules have a basic configuration setup for them read: one test hostgroup per puppet module and a simple puppet report check for the launched test VM to verify they are configured right Integration tests done for some (e.g. base modules) 12/5/2014 ITTF on Continuous Integration 21
How it works Launch a test VM that includes the changed module Check its reports Run the module's tests if they exist Kill the machine Report accordingly to Jira 12/5/2014 ITTF on Continuous Integration 22
Tests like what? Mysql: can I create a database & table, insert data and run queries? Shibboleth: can I access a test web server with X credentials? AFS: Is my private data accessible by me and only me? 12/5/2014 ITTF on Continuous Integration 23
Getting involved If you maintain a puppet module you should probably start testing Quickstart templates exist on our master for both static puppet testing and dynamic testing Contact ai-config-team if you need some hands-on support 12/5/2014 ITTF on Continuous Integration 24
Dev'ing Gitlab merge requests Already running a small test suite for ai-tools Refactoring as much as possible using a new brilliant workflow plugin 12/5/2014 ITTF on Continuous Integration 25
Upcoming 'workflow-plugin' In development by the cloudbees team Groovy DSL Parallel flows Sequential staging when needed Grab slave nodes, allocate workspaces... Pause and demand input to resume Resume execution after crash Main concept is to fit everything in one job 12/5/2014 ITTF on Continuous Integration 26
Upcoming 'workflow-plugin' Should allow for much bigger freedom in the CI pipeline Multiple modules and architectures tested per CRM ticket Hopefully all major features will be available in open source If you're already using Jenkins you should probably check it out 12/5/2014 ITTF on Continuous Integration 27
Refs http://www.jenkins-ci.org https://servant.cern.ch http://configdocs.web.cern.ch/configdocs/changes/cr m.html#using-the-continuous-integration-workflow https://github.com/jenkinsci/workflow-plugin 12/5/2014 ITTF on Continuous Integration 28
2/3 A Jenkins Platform Creating Jenkins Masters and Slaves in CERN Forge By Terje Ness Andersen
Motivation Make getting started easier, building on AI experiences Take care of common tasks Single Sign-On Backup SSH Keys Kerberos authentication 12/5/2014 ITTF on Continuous Integration 30
Current prototype Pre-configured Jenkins Master with Slaves Single Sign-On with custom ACL Jenkins configuration and job data stored on persistent volume SSH Key + Kerberos identity for Master+Slaves 12/5/2014 ITTF on Continuous Integration 31
Creation form in CERN Forge Instance Title Access Control List for Single Sign-On Slaves how many, with which size and OS? May be edited after creation 12/5/2014 ITTF on Continuous Integration 32
Creation view in CERN Forge 12/5/2014 ITTF on Continuous Integration 33
Detail view in CERN Forge 12/5/2014 ITTF on Continuous Integration 34
Architecture One VM per Jenkins Master and Slave Jenkins Masters GUI accessible via jenkins.cern.ch Apache frontend reverse proxies to Master Slaves communicate with Jenkins Master directly over HTTPS or SSH 12/5/2014 ITTF on Continuous Integration 35
Jenkins Master 12/5/2014 ITTF on Continuous Integration 36
Architecture Single Sign-On master-1 jenkins.cern.ch frontend-1 frontend-2 master-2 slave-xx master-3 12/5/2014 ITTF on Continuous Integration 37
Configuration ci/jenkins_frontend ci/jenkins_master ci/jenkins_slave Configured with CERN Forge REST API Read for each Puppet Run Uptime and validity critical 12/5/2014 ITTF on Continuous Integration 38
Jenkins Master Initial access control in Jenkins with instance owner as administrator. No administration constraints. Plugins defined in CERN Forge automatically installed. 12/5/2014 ITTF on Continuous Integration 39
Jenkins Slaves Built-in slaves defined in CERN Forge Hosted and configured by us SLC5, SLC6 or CentOS7 Automatically set up in Master User-provided slaves more flexibility: Connect your own slaves Manually set up in master 12/5/2014 ITTF on Continuous Integration 40
Authentication SSH Key installed for Master and built-in slaves Kerberos ticket using machine identity for every Master and built-in slaves Information exposed on info page protected by Single Sign-On 12/5/2014 ITTF on Continuous Integration 41
jenkins.cern.ch/info/<title> 12/5/2014 ITTF on Continuous Integration 42
To-Do Backup of the Jenkins Master volumes Documentation with examples Example jobs in Jenkins Master World or CERN accessible Slave configuration presets 12/5/2014 ITTF on Continuous Integration 43
3/3 CI challenges and future outlook By Alexandre Lossent
Challenge 1: resource efficiency A number of standard slave configurations are provided Within the central Jenkins Openstack project 1 VM / slave Challenge 1: low resource efficiency Depending on the software project, most slave VMs may be doing nothing most of the time How to optimize this? 12/5/2014 ITTF on Continuous Integration 46
Resource efficiency: solutions? Apache Mesos + Docker Slaves running inside Docker containers Pool of VMs in an Apache Mesos cluster to run the Docker containers Enables a clean, fresh slave for every job User-provided Docker images possible with various OS's and environments 12/5/2014 ITTF on Continuous Integration 47
Challenge 2: orchestration Jenkins works with multiple slaves built-in slaves implemented as Puppetmanaged Openstack VMs Master and slaves currently created by hand Initial creation Add/remove slaves for a given project reset to a clean environment How to automate slave provisioning? 12/5/2014 ITTF on Continuous Integration 48
Orchestration: solutions? Custom orchestration workflow and tools Possibly on top of Openstack Heat if available Off-the-shelf orchestration tools for Openstack + Puppet E.g. Cloudify 12/5/2014 ITTF on Continuous Integration 49
Challenge 3: small projects From IT PoW 2015 summary: [ ] ~90 FTE s were involved in software development in one form or another, with many small < 1 FTE projects. Most large software projects using CI already in one form or another Examples in IT, GS, EN, PH, TE, BE TeamCity, Bamboo, Travis CI, Drone.io, Jenkins Challenge 3: How to get small projects (<1 FTE) on board the CI boat? 12/5/2014 ITTF on Continuous Integration 50
Hosted CI services Some examples of popular services for projects hosted on GitHub BuildHive.CloudBees.com (Jenkins) Travis-CI.org Drone.io How do they attract small projects? Zero or minimal configuration effort Zero or minimal cost/build A finite list of supported languages to build+test (Java, C++, PHP ) A list of supported PaaS targets for automated deployment (Google Apps, Heroku, Openshift ) 12/5/2014 ITTF on Continuous Integration 51
Small projects: solutions? Low cost of entry Enabling CI on a project should be a couple clicks away Low maintenance Little to no configuration Automatically detect project type/language Low cost in terms of resources resource efficiency gets even more important Well-defined target platforms Save time and effort by automating application deployment Compromise: no 100% coverage of projects 12/5/2014 ITTF on Continuous Integration 52
Beyond CI These 3 challenges have a scope larger than just Jenkins Docker, orchestration tools, well-defined hosting environments (probably) of interest for a lot of teams PaaS in particular has a really interesting paradigm Consider your typical small web application project 12/5/2014 ITTF on Continuous Integration 53
Typical small web app WebApp instance 1 WebApp instance 2 DNS LB User DBoD 12/5/2014 ITTF on Continuous Integration 54
Just within PES... ~20 typical web apps based on standard platforms From Batch monitoring to Configuration Management tools to Version Control UIs... Django, Ruby on Rails, Perl, Drupal, Tomcat Mobilising >30 VMs Each of them a unique setup 12/5/2014 ITTF on Continuous Integration 55
Typical process Set up a couple VMs to host the app Configuration: Puppet? Ad hoc? Maybe set up a DB Set up DNS Load Balancing Set up SSO Version control? CI/Testing? Long-term VM/OS maintenance... 12/5/2014 ITTF on Continuous Integration 56
Can we do better than this? PaaS Demo: CI in Openshift 12/5/2014 ITTF on Continuous Integration 57
Interesting properties of PaaS Low cost of entry Easily provision a few instances of a given web hosting environment, plus a database Resource efficiency Several instances desired for redundancy, yet each with a small load Low maintenance cost for devs Delegate OS and security hotfix management Trivially add CI to the app 12/5/2014 ITTF on Continuous Integration 58
Typical web app with CI WebApp instance 1 WebApp instance 2 DNS LB User Automatic deployment CI Run tests DBoD Git Dev 12/5/2014 ITTF on Continuous Integration 59
Options to get there? PaaS off-the-shelf solutions Openshift Cloudfoundry Smart orchestration of existing infrastructure and tools Shared Puppet modules, DNSLB, Openstack, SSO, central git service, Jenkins, DBoD Cloudify? 12/5/2014 ITTF on Continuous Integration 61
Conclusion Proven utility of CI in the context of Configuration Management Opportunity to offer CI to a broader audience A must have for all dev projects CI project facing challenges with broad scope Opportunity to improve app development process and hosting in general 12/5/2014 ITTF on Continuous Integration 62
12/5/2014 ITTF on Continuous Integration 63