System Management. Leif Nixon. a security perspective 1/37

Transcription

1 1/37 System Management a security perspective Leif Nixon

2 2/37 System updates Should we ever update the system? Some common update strategies: 1. If it works, don t touch it! 2. We pick and choose the most important updates. 3. We apply all updates.

3 3/37 System updates Discovering the need to update Watch vendor announcement lists Watch general security lists Be part of a community that can afford to watch more specific resources

4 /37 System updates Triage Do we care? Useful scale: 1. Update later (scheduled or together with more important stuff) 2. Update as soon as we get an update in our preferred format 3. Take care of it right now in one way or another!

5 5/37 System updates Triage Do we care? Caution: Important for the vendor may still mean Critical for you! Most security advisories are not tuned for our environment, with many general shell users. A local root exploit is critical to us!

6 6/37 System updates Updating the triage Generally, things get more broken with time.

7 6/37 System updates Updating the triage Generally, things get more broken with time. Sometimes: lucky breaks in the other direction; Ah, it turns out the bug is only triggered if you use feature X We found this simple workaround

8 7/37 System updates Problems getting updates Waiting for repackagers Example: CentOS Waiting for vendor solutions Example: proprietary filesystem drivers Figuring out stuff built from source at site Example: a lot of big scientific packages

9 8/37 System updates Solutions for getting updates If it hurts, don t do it Limit dependency on turn-key solutions, third-party dependencies as far as possible. Make vendors understand our pain. Again and again. Until they understand. Prepare for building own packages Example: For RHEL/CentOS set up mock environment, do test builds of likely/hard components (kernel, glibc,...)

10 9/37 System updates Solutions for getting updates Document source builds so they can be redone by others Automatic build scripts/systems may help but not if too complicated! Make sure you have the credentials, licenses etc needed to download vendor updates when you need them.

11 0/37 Workarounds Trust, but verify Can we test that it works? Do we trust it? Always remember that even if one particular exploit fails, you may still be vulnerable...

12 11/37 Workarounds Systemtap to the rescue Systemtap is often useful for working around kernel vulnerabilities. Example: to block the CVE vulnerability in the perfmonitor subsystem: probe kernel.function("sys_perf_event_open") printf("sys_perf_event_open DENIED!\n"); $attr_uptr = 0 }

13 12/37 How to deploy updates Updating login nodes and system servers Reboot not needed Updates can often be applied right away. Make sure enough stuff is restarted! Do you know about /usr/bin/needs-restarting? Reboot needed Multiple login servers help. Multiple redundant system servers help too. Otherwise, make sure a quick reboot does not cause user problems (lost jobs, failed file operations,... )

14 13/37 How to deploy updates Updating worker nodes Reboot not needed Updates may be applied right away, but do you care about OS jitter? Can we do work in the job epilog?... are you sure the update persists? (more on this later)

15 14/37 How to deploy updates Updating worker nodes Reboot needed Automate rolling updates (rebooting as soon as current job is done)! Keep users out of nodes they do not run jobs on. Can we accept the risk that users can login to nodes that are not yet updated? If not, shut down user login access to nodes.

16 15/37 No workaround, no update? Decide beforehand when it will be appropriate to shut down access. Get management acceptance for this.

17 15/37 No workaround, no update? Decide beforehand when it will be appropriate to shut down access. Get management acceptance for this. Levels: 1. No new logins accepted, but do not kick out those who are logged in. 2. No logins accepted, logged in users kicked out, jobs keep running. 3. Jobs killed, too.

18 16/37 No workaround, no update? Keep users and management informed! We have temporarily blocked logins while we are investigating how to secure the system against a serious security vulnerability that was released today easily becomes I dunno, I heard they shut down the system because it was hacked or something.

19 17/37 Configuration management A consistent system configuration over nodes and servers is important for security, but also for functionality and performance.

20 18/37 Configuration management Node installation Special consideration for compute nodes: When and how do we (re-)install them? 1. Manually installed node image, use that when booting 2. Scripted install (kickstart etc) for making node image, use that when booting 3. Scripted install when needed, reboot from disk otherwise 4. Scripted install on every boot

21 9/37 Configuration management Node installation Node reinstallation on every reboot: Good: Nodes are always clean. Bad: Nodes already zapped when you want to do forensics.

22 20/37 Configuration management Deployment at scale Letting thousands of servers wget/rsync from a few system servers might not work. Solutions: Bittorrent? Multicast?

23 21/37 Configuration management Keeping things consistent How do we make sure we do not forget vital configuration? 1. Checklists 2. Scripts 3. Configuration management tools Ansible, Cfengine, Chef, Puppet, Quattor,...

24 21/37 Configuration management Keeping things consistent How do we make sure we do not forget vital configuration? 1. Checklists 2. Scripts 3. Configuration management tools Ansible, Cfengine, Chef, Puppet, Quattor,... Warning: Abstracting too much may make system administration harder. Strive for the right balance!

25 22/37 Configuration management Keeping things consistent Then add version control on top of this. Now, you also know how it was four months ago! Log your actions. Still useful, even with configuration management tools (but may partly be the commit log). May be as simple as a date-ordered text file on the system server.

26 23/37 Configuration management Keeping things consistent Package your own tools, scripts etc. Example: Instead of some slightly different scripts in /root/bin on four clusters, we might aim for versioned RPM package on internal repo server, with source in git.

27 23/37 Configuration management Keeping things consistent Package your own tools, scripts etc. Example: Instead of some slightly different scripts in /root/bin on four clusters, we might aim for versioned RPM package on internal repo server, with source in git. But again, don t overdo this...

28 24/37 Configuration management Node health checking Running health checking scripts in prolog/epilog might not be all that security related, but it saves a lot of other trouble... We can add test for security problem X fixed if we want.

29 25/37 Compartmentalization The waterfall model of trust Be aware of the direction of the trust flow. For example: OK: desktop system server compute node OK: desktop system server login node

30 5/37 Compartmentalization The waterfall model of trust Be aware of the direction of the trust flow. For example: OK: desktop system server compute node OK: desktop system server login node Bad: desktop login node system server Bad: login node console infrastructure server

31 6/37 Compartmentalization The waterfall model of trust Explain reasoning to all system staff. Yes, it may be inconvenient at times. Yes, it may save your cluster one day. Enforce using account filters, firewalls, etc.

32 27/37 Compartmentalization Separate servers Users should only have access to the login nodes and, possibly, their allocated worker nodes. Keep system servers separate from login nodes. On large systems, separate system servers more if possible. Example: File servers may require kernel versions with known local root exploits for weeks or forever. Restrict access to them!

33 28/37 Compartmentalization Separate credentials Do not use the same password or similar for different levels in the waterfall. Do not use the same root password on different clusters. Goal: Nothing you can steal at a lower level should gain you access at a higher level. But it s hard to fully reach this.

34 29/37 Compartmentalization Separate credentials Example: Only staff can login to system servers. Good. But staff homedirs are mounted on login nodes and compute nodes. If you get root on them, you can become a staff member, change.profile, get to run code on system server (or boobytrap ssh, su,... )

35 30/37 Compartmentalization Separate credentials Be careful with single sign-on! Example: Library password LiU password Personal certificate Supercomputer access!

36 31/37 Hardening There are a lot of things we can do. Some may conflict with ease-of-use, some not.

37 32/37 Hardening suid stripping Example: NSC antisuid runs from cron Whitelist of known binaries that are allowed to be setuid root ping, sudo... Blacklist of known binaries that should have their suid bit stripped mount... Any other suid binaries found? Sound the alarm!

38 33/37 Hardening File system flags Whenever possible, mount file systems: read-only nosuid, nodev root squash norelatime performance vs. forensic abilities

39 34/37 Hardening Module loading Many kernel exploits depend on being able to load some obscure module. Questions to consider: Can we turn off module loading completely? Can we at least turn off autoloading of modules?

40 35/37 Hardening Security frameworks SELinux, grsecurity... NSC doesn t do this. Any success stories? Introducing third-party dependencies may make updating harder.

41 36/37 Hardening Help the users proactively Examples of things you can do periodically: Check for really bad filesystem permissions Check for bad SSH usage Unencrypted private keys Known bad keys in authorized_keys (remember the Debian Debacle?)

42 7/37 And much, much more... We still haven t covered things like log management, log analysis, intrusion detection...

43 7/37 And much, much more... We still haven t covered things like log management, log analysis, intrusion detection so here comes another 40 slides!

44 7/37 And much, much more... We still haven t covered things like log management, log analysis, intrusion detection just kidding!

45 7/37 And much, much more... We still haven t covered things like log management, log analysis, intrusion detection just kidding! Questions? Opinions? Protests?