Configuration Management Change Management, and Culture Management

Transcription

1 Configuration Management Change Management, and Culture Management HEPiX Fall Thursday October 16 4:00pm James Pryor - [email protected] RHIC and ATLAS Computing Facility at Brookhaven National Laboratory

2 Configuration Management Change Management, and Culture Management Past Present Future Plans & Desires Credits / Discussion 2

3 Past Many ways to do OS/application deployment and configuration OS deploy: a single PXE/kickstart server in 2007.Lots of kickstart files. At best 5:1, worst 1:1. We used a post install RPM file with base level config files & scripts Started with Cfengine in 2008 on RH Linux and Solaris Used it for very basic OS configuration and some application configuration 3

4 Past: 2010 Identified some problems local & global. Kickstart file is installation only. Once installed, changes over time make unique configurations. We found Cfengine 2.x and Cfengine 3.x not ideal On a wider scope: We (humans) don't scale. Separate pools of knowledge. Files, internal web, internal mail, our minds, external web. More than one way to create/fix/diagnose it. Errors & issues are hard to debug/replicate. We can not do it all or remember it all. 4

5 We can not do it all. We don't have super powers. Shawn Hoke Attribution-NonCommercial-NoDerivs 2.0 Generic (CC BY-NC-ND 2.0) 5

6 Past: Change Management Policy & procedures (can be formal as ITIL, or as informal as DevOps) used to control changes and keep a historical change record made to production/development/test systems Uncontrolled change can work, but will often cause self inflicted problems and future firefighting episodes & upgrade nightmares Without it, servers/applications become like snowflakes: they start out identical, but over time, configuration drift eventually makes each one unique. 6

7 They are pretty to look at but hard to manage. Alexey Kljatov - Creative Commons (CC BY-NC 2.0)

8 Past: Change Management Increase agility. Do More with less. Shorten time to solution. Standardize! We want to stop duplicating work and effort. Almost everyone is using change management. Align ourselves to be congruent with other entities (Labs, Univ, Corp) so that we can build upon their success. Follow proven methods of improvement and change management to shift staff time from perpetual reactive mode (firefighting) to more proactive work (fire prevention) No unauthorized changes, no "cowboy" or "superhero" type behavior tolerated 8

9 Past: 2010 Moved to Puppet for configuration management: easy to learn, facts, modular, idempotence, reporting. Intro level training in Aug 2010 Fall 2010, we chose Cobbler for Linux OS deployment via PXE/Kickstart. CLI & Web manages distros, systems, and repos. Modularity through variables, templates, and profiles. Selected GLPI for asset management, and FusionInventory Agent to collect server details 9

10 Past: 2011 to 2012 Convert kickstarts into Cobbler single template to support just about all use cases, and convert the RPM file post-install configuration shell scripts into Puppet code. Developed front-end Perl web scripts to manage SSL CA certs, and act as change management 'gateway' to back-end git/cgit Puppet External Node Classifier: GLPI, Puppet Dashboard, Linux worker node pool custom inventory DB Created a foundational base class that encapsulates the desired default configuration, then worked on managing services with puppet. Started working on Desktop Centralized Management 10

11 This flowchart is from a presentation made in late

12 Cobbler System view - racf-min.ks is our shared templated kickstart 12

13 GLPI Server details. Note the circled Custom tab. 13

14 GLPI Custom tab details. Note the circled base class. 14

15 Base class is 'community property' where we build the foundation for infrastructure servers. Code edited/resized for screenshot 15

16 This is the linuxfarm group's workernode class 16

17 This shows we have one client host cert that need to be signed. 17

18 I need to clean a client host cert, so I can search for that host with regex support. 18

19 It is cleaning one host cert and removing and exported resources tied to that host. 19

20 Present: Make a change to production pryor@mydesktop (production) ~/gitree/catalog/common/cvesecurity/manifests $ git push origin production <trimmed> remote: diff-tree: remote: : eb0bc... 7bf289b... M common/cvesecurity/manifests/cve_2014_7169.pp remote: Merge results: remote: Updating ce24fcc..d8a7860 remote: Fast-forward remote: common/cvesecurity/manifests/cve_2014_7169.pp 1 + remote: 1 file changed, 1 insertion(+) remote: Note: Your updates to the production branch are waiting for approval in branch: pending-production-pryor-ce24fcc t201915utc remote: remote: error: hook declined to update refs/heads/production To [remote rejected] production -> production (hook declined) error: failed to push some refs to ' 20

21 Puppet Approval Change Management Portal showing my commit to production 21

22 Diff of my commit as seen in cgit 22

23 Do you want to merge this? 23

24 Puppet Change Approval Committe. Note the regex under the Authorization column. 24

25 RACF Puppet Catalog. The regex from the previous slide matches up with the directories (in blue). 25

26 Merge results 26

27 Approval scripts also record who approved what for auditing purposes 27

28 Summary of all log messages making up the branch merge in cases where there are many commits in that branch 28

29 Present: Reporting, Performance Metrics Puppet Dashboard is deprecated. We now use The Foreman and PuppetDB for puppet client run reporting. The Foreman is a RHEL 6.5 VM on the RHEV cluster, and assigned 8 vcpus and 8GB RAM. A single Puppet Master server: Dell PowerEdge R610 with Dual CPU Intel Xeon X GHz, 96GB RAM, RAID10 900GB, RHEL 6.5 It runs just about everything: Apache, Perl Puppet CA, Perl Puppet Approval, git/cgit, PuppetDB (w/ PostgreSQL), GLPI (w/ MySQL) 29

30 30

31 Puppet Client agent check-in method & interval The Foreman VM on RHEV PostgreSQL Puppet Master Dell PowerEdge R610 Dual CPU Intel Xeon X GHz 96GB RAM, RAID10 900GB,RHEL 6.5 Apache, Perl Puppet CA, Perl Puppet Approval, git/cgit, PuppetDB (w/ PostgreSQL) GLPI (w/ MySQL) Manual check_puppet.py run without Puppet daemon Puppet daemon check-in once per hour Linux farm worker nodes Grid, Infrastructure, Storage, Cloud hosts under 600 machines Farm Infrastructure, DB 31

32 Present: Load test & Metrics We tested our Puppet master with client agents checking in at about 1Hz. PuppetDB reports that we have over 2500 checking-in during this test, which ran a little over an hour. Load (top command output) as seen at about 12 minutes after starting a manual puppet agent run (14:45) on the Linux Farm worker nodes. top - 14:57:23 up 3:55, 2 users, load average: 4.22, 4.22, 3.03 Tasks: 551 total, 4 running, 547 sleeping, 0 stopped, 0 zombie Cpu(s): 20.9%us, 2.1%sy, 0.0%ni, 76.8%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st Mem: k total, k used, k free, k buffers Swap: k total, 0k used, k free, k cached PID USER puppet puppet puppet puppetdb puppet puppet puppet postgres postgres PR NI VIRT RES SHR S %CPU %MEM 0 302m 195m 3404 S m 183m 3404 S m 254m 3404 R g 1.3g 12m S m 244m 3404 R m 169m 3404 S m 199m 3400 S g 575m 569m S g 670m 662m S TIME+ 4: : : : : : : : :14.15 COMMAND ruby ruby ruby java ruby ruby ruby postmaster postmaster 32

33 Linux farm manual run started approx 14:45. 33

34 In a bit more than an hour, we saw between 40% - 60% CPU with about 2500 agents checking in. 34

35 PuppetDB metrics during the load test 35

36 PuppetDB metrics during the load test 36

37 PuppetDB metrics during the load test 37

38 Present: Puppet Code Statistics Project name: RACF Puppet Catalog Generated: :05:07 (in 6 seconds) Generator: GitStats (version ), git version Report Period: :54:30 to :59:44 Age: 1375 days, 877 active days (63.78%) Total Files: 2620 Total Lines of Code: ( added, removed) Average file size: bytes Total Commits: (average 15.3 commits per active day, 9.7 per all days) Authors: 31 (average commits per author) 146 modules, 431 different.pp files, totaling nearly 34k lines of Puppet code. 38

39 Present: Puppet Code Statistics 39

40 Present: Puppet code style guide In Aug & Sept of 2014, we used puppet-lint and Geppetto to update all our Puppet code to comply with the Puppet Labs's suggested style guide. Style guide is now enforced on push to the Git server via puppet-lint puppet-lint --with-context --no-80chars-check init.pp WARNING: class not documented on line 1 ERROR: trailing whitespace found on line 66 } ^ WARNING: indentation of => is not properly aligned on line 81 source => $allow_from, ^ WARNING: unquoted file mode on line 53 mode => 644, ^ 40

41 41

42 Present: Culture and CM Adoption Discovered that changing servers is easy. Changing people's work flow and ultimately their minds is much harder. At first mixed reception and some resistance. Most people within our group adopted this work flow & toolset. Others not at all. Attribution 2.5 Generic (CC BY 2.5) 42

43 Present: Puppet Use / Culture Management Had some accidental & potentially dangerous commits to production branch. For the Change Management Approval gateway, implemented a self-approve delay, mandatory scroll & approval authorizations Never manually change production servers or commit untested code to production. Use test environments & servers. We host a tree of Puppet code base ( common ) & is now shared and used upstream by the Physics and IT departments. 43

44 If your infrastructure is now done in code, do you test your code? Or do you just push it to production and see if it works? 44

45 Dos Equis Man The Most Interesting Man in the World is a character & property of Cervecería Cuauhtémoc-Moctezuma. Used without permisson on assumption of Fair Use / Parody 45

46 He is not someone to admire. Don't be like him. You must test your code. Dos Equis Man The Most Interesting Man in the World is a character & property of Cervecería Cuauhtémoc-Moctezuma. Used without permisson on assumption of Fair Use / Parody 46

47 Future Plans and Desires We look to adopt software development practices for our Puppet code: smoke testing, unit testing, acceptance testing Automatic testing (Continuous Integration) system with Jenkins CI tool, to manage our Puppet testing process. It runs Puppet on a pool of RHEV VMs,and all pending changes to production must pass this validation process before they can be approved and merged into production. See talk on this topic at future HEPiX and/or CHEP Use The Foreman beyond just reporting: as both ENC & as OS provisioner. 47

48 Future Plans and Desires Work toward a community shared Puppet code base beyond RACF/Physics/Lab ITD. This is desirable but is at least 1 2 years away from being realized. Puppet with Hiera: a hierarchical data store keeps sitespecific data out of your manifests. Avoid repetition (duplicating similar blocks of modular code), and use public Puppet modules. When used, don t need to edit the code, just put the necessary data in Hiera. Requires a rewrite/refactor of our code. This is a non-trivial project. 48

49 Future Plans and Desires Mcollective: a framework to build server orchestration, parallel job execution, on clusters of servers. Not simply a fancy SSH "for loop", but provides granularity and reporting. Integrate Monitoring and Puppet: rewrite existing nagios puppet class to support use of exported resources. Both the target node to be monitored and the Nagios server would execute Puppet code in a sort of conversation: "Hey Nagios server. I'm a node and have a new service. You need to monitor it." Then it would be monitored. 49

50 Credit / Questions Dr. Jason A. Smith, Mizuki Karasawa, John S. De Stefano Jr. William Strecker-Kellog, Christopher Hollowell, James Pryor "Bernard De Chartres used to compare us to [puny] dwarfs perched on the shoulders of giants. He pointed out that we see more and farther than our predecessors, not because we have keener vision or greater height, but because we are lifted up and borne aloft on their gigantic stature." John of Salisbury