A central continuous integration platform

Transcription

1

2 A central continuous integration platform Agile Infrastructure use case and future plans Dec 5th, 2014

3 1/3 The Agile Infrastructure Use Case By Stefanos Georgiou

4 What? Development practice Build better quality software, faster Developers integrate their work frequently Applying quality control with each integration Verified by an automated process Tests tests tests 12/5/2014 ITTF on Continuous Integration 4

5 You mean just testing compilation? Not quite.. Compiling Static & Dynamic Testing Unit & Integration Testing Code reviewing Deploying Documenting 12/5/2014 ITTF on Continuous Integration 5

6 Why? Postmortem bug-fixing is expensive Automate repetitive manual processes Maintain cohesive software and quality code base Visibility aggregated test data are showing trends No undeployable software It works on my machine 12/5/2014 ITTF on Continuous Integration 6

7 How? 12/5/2014 ITTF on Continuous Integration 7

8 Who? Top CI server in use Over 1000 Plugins Check out from your VCS automatically, run your tests, code coverage or static analysis, notify ( , rss, jabber) 12/5/2014 ITTF on Continuous Integration 8

9 What's happening in Configuration Management? Our own jenkins master available at servant.cern.ch SSO & egroup-based job ownership support Daily building VMs and testing puppet modules CRM workflow runs here Not exactly pure CI QA process per change, multiple repos Getting there 12/5/2014 ITTF on Continuous Integration 9

10 12/5/2014 ITTF on Continuous Integration 10

11 CRM ticket process Shared module needs a change People using it probably come from different projects You can't just merge code to master Create CRM ticket in JIRA to notify for the change and wait 12/5/2014 ITTF on Continuous Integration 11

12 Previously in the CRM Workflow Write your puppet code inside a feature branch Test it yourself Create a ticket on the CRM JIRA project Merge to QA Hold for a week (or two) Any complaints? Merge to master 12/5/2014 ITTF on Continuous Integration 12

13 Rather do it like this Develop inside a feature branch 12/5/2014 ITTF on Continuous Integration 13

14 Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project 12/5/2014 ITTF on Continuous Integration 14

15 Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project Tick the box 12/5/2014 ITTF on Continuous Integration 15

16 Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project Tick the box Jenkins will run tests from your feature branch and merge it to QA for you 12/5/2014 ITTF on Continuous Integration 16

17 Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project Tick the box Jenkins will run tests from your feature branch and merge it to QA for you Hold for a week (or two) 12/5/2014 ITTF on Continuous Integration 17

18 Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project Tick the box Jenkins will run tests from your feature branch and merge it to QA for you Hold for a week (or two) Broken? 12/5/2014 ITTF on Continuous Integration 18

19 Rather do it like this Develop inside a feature branch Create a ticket on the CRM JIRA project Tick the box Jenkins will run tests from your feature branch and merge it to QA for you Hold for a week (or two) Broken? Jenkins will merge to master on proposed date after final test, assuming no problems found 12/5/2014 ITTF on Continuous Integration 19

20 Feedback from the build 12/5/2014 ITTF on Continuous Integration 20

21 What's being tested right now? Most modules have a basic configuration setup for them read: one test hostgroup per puppet module and a simple puppet report check for the launched test VM to verify they are configured right Integration tests done for some (e.g. base modules) 12/5/2014 ITTF on Continuous Integration 21

22 How it works Launch a test VM that includes the changed module Check its reports Run the module's tests if they exist Kill the machine Report accordingly to Jira 12/5/2014 ITTF on Continuous Integration 22

23 Tests like what? Mysql: can I create a database & table, insert data and run queries? Shibboleth: can I access a test web server with X credentials? AFS: Is my private data accessible by me and only me? 12/5/2014 ITTF on Continuous Integration 23

24 Getting involved If you maintain a puppet module you should probably start testing Quickstart templates exist on our master for both static puppet testing and dynamic testing Contact ai-config-team if you need some hands-on support 12/5/2014 ITTF on Continuous Integration 24

25 Dev'ing Gitlab merge requests Already running a small test suite for ai-tools Refactoring as much as possible using a new brilliant workflow plugin 12/5/2014 ITTF on Continuous Integration 25

26 Upcoming 'workflow-plugin' In development by the cloudbees team Groovy DSL Parallel flows Sequential staging when needed Grab slave nodes, allocate workspaces... Pause and demand input to resume Resume execution after crash Main concept is to fit everything in one job 12/5/2014 ITTF on Continuous Integration 26

27 Upcoming 'workflow-plugin' Should allow for much bigger freedom in the CI pipeline Multiple modules and architectures tested per CRM ticket Hopefully all major features will be available in open source If you're already using Jenkins you should probably check it out 12/5/2014 ITTF on Continuous Integration 27

28 Refs m.html#using-the-continuous-integration-workflow 12/5/2014 ITTF on Continuous Integration 28

29 2/3 A Jenkins Platform Creating Jenkins Masters and Slaves in CERN Forge By Terje Ness Andersen

30 Motivation Make getting started easier, building on AI experiences Take care of common tasks Single Sign-On Backup SSH Keys Kerberos authentication 12/5/2014 ITTF on Continuous Integration 30

31 Current prototype Pre-configured Jenkins Master with Slaves Single Sign-On with custom ACL Jenkins configuration and job data stored on persistent volume SSH Key + Kerberos identity for Master+Slaves 12/5/2014 ITTF on Continuous Integration 31

32 Creation form in CERN Forge Instance Title Access Control List for Single Sign-On Slaves how many, with which size and OS? May be edited after creation 12/5/2014 ITTF on Continuous Integration 32

33 Creation view in CERN Forge 12/5/2014 ITTF on Continuous Integration 33

34 Detail view in CERN Forge 12/5/2014 ITTF on Continuous Integration 34

35 Architecture One VM per Jenkins Master and Slave Jenkins Masters GUI accessible via jenkins.cern.ch Apache frontend reverse proxies to Master Slaves communicate with Jenkins Master directly over HTTPS or SSH 12/5/2014 ITTF on Continuous Integration 35

36 Jenkins Master 12/5/2014 ITTF on Continuous Integration 36

37 Architecture Single Sign-On master-1 jenkins.cern.ch frontend-1 frontend-2 master-2 slave-xx master-3 12/5/2014 ITTF on Continuous Integration 37

38 Configuration ci/jenkins_frontend ci/jenkins_master ci/jenkins_slave Configured with CERN Forge REST API Read for each Puppet Run Uptime and validity critical 12/5/2014 ITTF on Continuous Integration 38

39 Jenkins Master Initial access control in Jenkins with instance owner as administrator. No administration constraints. Plugins defined in CERN Forge automatically installed. 12/5/2014 ITTF on Continuous Integration 39

40 Jenkins Slaves Built-in slaves defined in CERN Forge Hosted and configured by us SLC5, SLC6 or CentOS7 Automatically set up in Master User-provided slaves more flexibility: Connect your own slaves Manually set up in master 12/5/2014 ITTF on Continuous Integration 40

41 Authentication SSH Key installed for Master and built-in slaves Kerberos ticket using machine identity for every Master and built-in slaves Information exposed on info page protected by Single Sign-On 12/5/2014 ITTF on Continuous Integration 41

42 jenkins.cern.ch/info/<title> 12/5/2014 ITTF on Continuous Integration 42

43 To-Do Backup of the Jenkins Master volumes Documentation with examples Example jobs in Jenkins Master World or CERN accessible Slave configuration presets 12/5/2014 ITTF on Continuous Integration 43

44 3/3 CI challenges and future outlook By Alexandre Lossent

45 Challenge 1: resource efficiency A number of standard slave configurations are provided Within the central Jenkins Openstack project 1 VM / slave Challenge 1: low resource efficiency Depending on the software project, most slave VMs may be doing nothing most of the time How to optimize this? 12/5/2014 ITTF on Continuous Integration 46

46 Resource efficiency: solutions? Apache Mesos + Docker Slaves running inside Docker containers Pool of VMs in an Apache Mesos cluster to run the Docker containers Enables a clean, fresh slave for every job User-provided Docker images possible with various OS's and environments 12/5/2014 ITTF on Continuous Integration 47

47 Challenge 2: orchestration Jenkins works with multiple slaves built-in slaves implemented as Puppetmanaged Openstack VMs Master and slaves currently created by hand Initial creation Add/remove slaves for a given project reset to a clean environment How to automate slave provisioning? 12/5/2014 ITTF on Continuous Integration 48

48 Orchestration: solutions? Custom orchestration workflow and tools Possibly on top of Openstack Heat if available Off-the-shelf orchestration tools for Openstack + Puppet E.g. Cloudify 12/5/2014 ITTF on Continuous Integration 49

49 Challenge 3: small projects From IT PoW 2015 summary: [ ] ~90 FTE s were involved in software development in one form or another, with many small < 1 FTE projects. Most large software projects using CI already in one form or another Examples in IT, GS, EN, PH, TE, BE TeamCity, Bamboo, Travis CI, Drone.io, Jenkins Challenge 3: How to get small projects (<1 FTE) on board the CI boat? 12/5/2014 ITTF on Continuous Integration 50

50 Hosted CI services Some examples of popular services for projects hosted on GitHub BuildHive.CloudBees.com (Jenkins) Travis-CI.org Drone.io How do they attract small projects? Zero or minimal configuration effort Zero or minimal cost/build A finite list of supported languages to build+test (Java, C++, PHP ) A list of supported PaaS targets for automated deployment (Google Apps, Heroku, Openshift ) 12/5/2014 ITTF on Continuous Integration 51

51 Small projects: solutions? Low cost of entry Enabling CI on a project should be a couple clicks away Low maintenance Little to no configuration Automatically detect project type/language Low cost in terms of resources resource efficiency gets even more important Well-defined target platforms Save time and effort by automating application deployment Compromise: no 100% coverage of projects 12/5/2014 ITTF on Continuous Integration 52

52 Beyond CI These 3 challenges have a scope larger than just Jenkins Docker, orchestration tools, well-defined hosting environments (probably) of interest for a lot of teams PaaS in particular has a really interesting paradigm Consider your typical small web application project 12/5/2014 ITTF on Continuous Integration 53

53 Typical small web app WebApp instance 1 WebApp instance 2 DNS LB User DBoD 12/5/2014 ITTF on Continuous Integration 54

54 Just within PES... ~20 typical web apps based on standard platforms From Batch monitoring to Configuration Management tools to Version Control UIs... Django, Ruby on Rails, Perl, Drupal, Tomcat Mobilising >30 VMs Each of them a unique setup 12/5/2014 ITTF on Continuous Integration 55

55 Typical process Set up a couple VMs to host the app Configuration: Puppet? Ad hoc? Maybe set up a DB Set up DNS Load Balancing Set up SSO Version control? CI/Testing? Long-term VM/OS maintenance... 12/5/2014 ITTF on Continuous Integration 56

56 Can we do better than this? PaaS Demo: CI in Openshift 12/5/2014 ITTF on Continuous Integration 57

57 Interesting properties of PaaS Low cost of entry Easily provision a few instances of a given web hosting environment, plus a database Resource efficiency Several instances desired for redundancy, yet each with a small load Low maintenance cost for devs Delegate OS and security hotfix management Trivially add CI to the app 12/5/2014 ITTF on Continuous Integration 58

58 Typical web app with CI WebApp instance 1 WebApp instance 2 DNS LB User Automatic deployment CI Run tests DBoD Git Dev 12/5/2014 ITTF on Continuous Integration 59

59 Options to get there? PaaS off-the-shelf solutions Openshift Cloudfoundry Smart orchestration of existing infrastructure and tools Shared Puppet modules, DNSLB, Openstack, SSO, central git service, Jenkins, DBoD Cloudify? 12/5/2014 ITTF on Continuous Integration 61

60 Conclusion Proven utility of CI in the context of Configuration Management Opportunity to offer CI to a broader audience A must have for all dev projects CI project facing challenges with broad scope Opportunity to improve app development process and hosting in general 12/5/2014 ITTF on Continuous Integration 62

61 12/5/2014 ITTF on Continuous Integration 63